Discussion for Home page

Christian Kulas — Wed, 27 Mar 2013 18:54:09 -0000

Parses PDF files of scientific articles based on naive bayes and sophisticated heuristics. The output is a XML file that contains the parsed data. Meta data is detected and marked as such.

The meta data contains the following elements:

Title
Authors
Abstract
Text
Headlines
Enumerations
References (Literature)

In the first step, the text elements are divided into blocks (similar to paragraphs) and after that, predictions for each element are made.

The project contains three runnable classes that can work on given PDFs in batch mode via threading:

a) BatchHeuristic: A parser that uses defined heuristics and rules. Especially applicable for articles with a broad set of layouts (e.g. PeDocs, http://www.pedocs.de/).
b) BatchHybrid: A parser that uses machine learning (Naive Bayes) to find the correct element.
c) ModelGenerator: Generates a training model, used by BatchHybrid, from given PDF and XML files. The training PDF Files must be in the same folder as the manually prepared XML files. The XML files do not have to be complete, but they must be in the correct order. An example XML file is also provided.

The whole project was written in Java and makes use of the library "PDFBox". The code is available under the Apache License 2.0.

It can be used in the following fashion:

a) java -Xms512M -Xmx2048M -classpath parser.jar de.yones.pdf.runnable.BatchHeuristic <pdf folder> <output folder> (<config file>)
b) java -Xms512M -Xmx2048M -classpath parser.jar de.yones.pdf.runnable.BatchHybrid <pdf folder> <output folder> <model file> (<config file>)
c) java -Xms512M –Xmx2048M –classpath parser.jar de.yones.pdf.runnable.ModelGenerator <training folder> <output model file> (<config file>)

Be aware of the large amount of resources that are required, therefore the heap size was dramatically increased in these examples.

WikiPage Home modified by Christian Kulas

Christian Kulas — Wed, 13 Mar 2013 18:59:23 -0000

Welcome to your wiki!

This is the default page, edit it as you see fit. To add a new page simply reference it within brackets, e.g.: [SamplePage].

The wiki uses Markdown syntax.

Christian Kulas

Recent changes to Home

Discussion for Home page

WikiPage Home modified by Christian Kulas