gamedevlists-general

[GD-General] Text file parsing

[Marc F. <mf...@pu...>]

Hello,

I'm currently writing a toolkit that will be used as a basis for my next
game engine. It will feature one or several editing tools for content
creation, as one of the objective is to be data-driven. In the process,
I plan to use some intermediate text-based file formats, such as ASE for
meshes (easy to export from any modelling packages), text
shaders/material files and so on.

My question is about the best way of writing file-parsers for these
files. I can see currently 2 options 

1) use some tools like Lex&Yacc 
- Pros : easy to use, fast development time, easy to make the grammar
evolve
- Cons : for now, I was unable to make Lex&Yacc create some nice C++
parser class that can be embedded in my tools.

2) write my own from scratch, maybe first writing a DumbParser class
that just provides with a bunch of basic string function, acting like a
Lexer and derive the different needed parsers from that, embedding the
grammar inside
- Pros : full control on the design and the implementation, in
comparison to the obfuscated code from Ley/Yacc
- Cons : more development time, grammar kind of hardcoded so not easily
maintainable, I'm not very experienced in writing text parsers :)

I wonder how people handle file parsing in their projects.
Thanks for your help.

-Marc
	

Re: [GD-General] Text file parsing

[Javier A. <ja...@py...>]

We use a structure very similar to ASE for our data definition files.
Writing the parser for it was about a couple hours.

If you're starting fresh, I would recommend using XML for structured text
files. You can use TinyXML to read them (use google to locate it).

Marc Fascia <mf...@pu...> wrote:
> Hello,
>
> I'm currently writing a toolkit that will be used as a basis for my
> next game engine. It will feature one or several editing tools for
> content creation, as one of the objective is to be data-driven. In
> the process, I plan to use some intermediate text-based file formats,
> such as ASE for meshes (easy to export from any modelling packages),
> text shaders/material files and so on.
>
> My question is about the best way of writing file-parsers for these
> files. I can see currently 2 options
>
> 1) use some tools like Lex&Yacc
> - Pros : easy to use, fast development time, easy to make the grammar
> evolve
> - Cons : for now, I was unable to make Lex&Yacc create some nice C++
> parser class that can be embedded in my tools.
>
> 2) write my own from scratch, maybe first writing a DumbParser class
> that just provides with a bunch of basic string function, acting like
> a Lexer and derive the different needed parsers from that, embedding
> the grammar inside
> - Pros : full control on the design and the implementation, in
> comparison to the obfuscated code from Ley/Yacc
> - Cons : more development time, grammar kind of hardcoded so not
> easily maintainable, I'm not very experienced in writing text parsers
> :)
>
> I wonder how people handle file parsing in their projects.
> Thanks for your help.
>
> -Marc


  Javier Arevalo
  Pyro Studios

RE: [GD-General] Text file parsing

[Awen L. <ali...@ed...>]

I warmly recommend the XML option as a text file. Very smart, even if you're
using only its simplest functionnalitites (tag enclosure and hierarchy save
you a lot of further format maintenance time). And maybe there are now some
recommendations ready concerning Scene/Mesh 3D. If it could help you...
Awen

-----Message d'origine-----
De : gam...@li...
[mailto:gam...@li...]De la part de
Javier Arevalo
Envoyé : mercredi 11 décembre 2002 12:17
À : gam...@li...
Objet : Re: [GD-General] Text file parsing


We use a structure very similar to ASE for our data definition files.
Writing the parser for it was about a couple hours.

If you're starting fresh, I would recommend using XML for structured text
files. You can use TinyXML to read them (use google to locate it).

Marc Fascia <mf...@pu...> wrote:
> Hello,
>
> I'm currently writing a toolkit that will be used as a basis for my
> next game engine. It will feature one or several editing tools for
> content creation, as one of the objective is to be data-driven. In
> the process, I plan to use some intermediate text-based file formats,
> such as ASE for meshes (easy to export from any modelling packages),
> text shaders/material files and so on.
>
> My question is about the best way of writing file-parsers for these
> files. I can see currently 2 options
>
> 1) use some tools like Lex&Yacc
> - Pros : easy to use, fast development time, easy to make the grammar
> evolve
> - Cons : for now, I was unable to make Lex&Yacc create some nice C++
> parser class that can be embedded in my tools.
>
> 2) write my own from scratch, maybe first writing a DumbParser class
> that just provides with a bunch of basic string function, acting like
> a Lexer and derive the different needed parsers from that, embedding
> the grammar inside
> - Pros : full control on the design and the implementation, in
> comparison to the obfuscated code from Ley/Yacc
> - Cons : more development time, grammar kind of hardcoded so not
> easily maintainable, I'm not very experienced in writing text parsers
> :)
>
> I wonder how people handle file parsing in their projects.
> Thanks for your help.
>
> -Marc


  Javier Arevalo
  Pyro Studios




-------------------------------------------------------
This sf.net email is sponsored by:
With Great Power, Comes Great Responsibility
Learn to use your power at OSDN's High Performance Computing Channel
http://hpc.devchannel.org/
_______________________________________________
Gamedevlists-general mailing list
Gam...@li...
https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
Archives:
http://sourceforge.net/mailarchive/forum.php?forum_id=557

Re: [GD-General] Text file parsing

[Achim M. <amu...@is...>]

Didn't try it myself, but http://www.irule.be/bvh/c++/olex/ pretends to
be: "olex is an easy to use LL(1) parser generator in C++, generating
C++ code."

-achim

> Marc Fascia wrote:
> 
> Hello,
> 
> I'm currently writing a toolkit that will be used as a basis for my
> next game engine. It will feature one or several editing tools for
> content creation, as one of the objective is to be data-driven. In the
> process, I plan to use some intermediate text-based file formats, such
> as ASE for meshes (easy to export from any modelling packages), text
> shaders/material files and so on.
> 
> My question is about the best way of writing file-parsers for these
> files. I can see currently 2 options
> 
> 1) use some tools like Lex&Yacc
> - Pros : easy to use, fast development time, easy to make the grammar
> evolve
> - Cons : for now, I was unable to make Lex&Yacc create some nice C++
> parser class that can be embedded in my tools.
> 
> 2) write my own from scratch, maybe first writing a DumbParser class
> that just provides with a bunch of basic string function, acting like
> a Lexer and derive the different needed parsers from that, embedding
> the grammar inside
> 
> - Pros : full control on the design and the implementation, in
> comparison to the obfuscated code from Ley/Yacc
> - Cons : more development time, grammar kind of hardcoded so not
> easily maintainable, I'm not very experienced in writing text parsers
> :)
> 
> I wonder how people handle file parsing in their projects.
> Thanks for your help.
> 
> -Marc
>

Re: [GD-General] Text file parsing

[Noel L. <ll...@co...>]

On Wed, 11 Dec 2002 12:16:35 +0100
Javier Arevalo <ja...@py...> wrote:

> If you're starting fresh, I would recommend using XML for structured text
> files. You can use TinyXML to read them (use google to locate it).

Yes, I was going to suggest XML as well. It's simple enough to replace
ini-like files, and complex enough that you can easily represent
hierarchical structures (very important for hierarchy/geometry
information).

I'm about to start looking into it more seriously for our next project,
so I don' t have any specific recommendations now.

I did use TinyXML for The Bidding Imp (http://www.convexhull.com/bidimp)
and it worked just fine. It has no data validation though (through DTD)
and that's something I think I would like to have, at least in the tool
side. Ideally, I'd like a parser that does both, and I can choose
whether to enable it or not. Still, it was certainly small enough (about
100KB) and it worked perfectly well just for saving/reading data.


--Noel
ll...@co...

Re: [GD-General] Text file parsing

[Ed S. <ed...@sw...>]

"Marc Fascia" <mf...@pu...> writes:
> I wonder how people handle file parsing in their projects.
Here's the excerpt from the gloriuos Scripting thread:

[Evan Bell]
  To avoid creating Yet Another Crappy Parser I used flex & bison to do 
  the hard work of creating the script compiler.

[Brian Hook]
  Eewww.  I'm leaning towards just using Lua as my syntax for, er, 
  everything, and letting it handle the heavy lifting.


Most people mentioned XML. But if you ask a Lua fan - he'll tell ya
that Lua is XML with a human face ;) 

-- Ed

Re: [GD-General] Text file parsing

[Ivan-Assen I. <as...@ha...>]

Text file parsingAnother point in favor of XML is the easiness with which
you can store Unicode characters in it, while keeping it
editable and viewable - just stuff UTF-8 or UTF-16 into an
XML file, set the encoding processing instruction and bingo,
your weird-language characters are there.

(Although, to be honest, we've had problems with encoding
for our XML-based localization tables, as some of the external
localizers (is that a word?) had trouble understanding
instructions like "find where in your software it says Encoding
and set it UTF-8". The only 100% idiot-proof weird-language safe
file format is Excel.)

Re: [GD-General] Text file parsing

[Ivan-Assen I. <as...@ha...>]

Text file parsingBy the way, if you are feeling TRULY adventurous, 
you might try Spirit (http://spirit.sourceforge.net).

It's a framework? library? metalibrary? for creating
parsers by describing the grammer directly within C++
code via a mindboggling amount of template trickery.

here's their "hello world" example - a parser which
takes a comma-separated list of floats and stuffs them
into a std::vector:

    bool
    parse_numbers(char const* str, vector<double>& v)
    {
        return parse(str,

            //  Begin grammar
            (
                real_p[append(v)] >> *(',' >> real_p[append(v)])
            )
            ,
            //  End grammar

            space_p).full;
    }

Translation: >> means "followed by", and is just whitespace in BNF;
the Kleene star *, means what it usually means, but by obvious
reasons has to be put IN FRONT of the subexpression.

Re: [GD-General] Text file parsing

[Tom S. <to...@pi...>]

> If you're starting fresh, I would recommend using XML for structured text
> files. You can use TinyXML to read them (use google to locate it).

    Also take a look at CMarkup ( http://www.firstobject.com/ ).  IMO it's
easier to use than all the other XML parsers i've tried... even ones in Java
and C#.  Tom

----- Original Message -----
From: "Javier Arevalo" <ja...@py...>
To: <gam...@li...>
Sent: Wednesday, December 11, 2002 5:16 AM
Subject: Re: [GD-General] Text file parsing


> We use a structure very similar to ASE for our data definition files.
> Writing the parser for it was about a couple hours.
>
> If you're starting fresh, I would recommend using XML for structured text
> files. You can use TinyXML to read them (use google to locate it).
>
> Marc Fascia <mf...@pu...> wrote:
> > Hello,
> >
> > I'm currently writing a toolkit that will be used as a basis for my
> > next game engine. It will feature one or several editing tools for
> > content creation, as one of the objective is to be data-driven. In
> > the process, I plan to use some intermediate text-based file formats,
> > such as ASE for meshes (easy to export from any modelling packages),
> > text shaders/material files and so on.
> >
> > My question is about the best way of writing file-parsers for these
> > files. I can see currently 2 options
> >
> > 1) use some tools like Lex&Yacc
> > - Pros : easy to use, fast development time, easy to make the grammar
> > evolve
> > - Cons : for now, I was unable to make Lex&Yacc create some nice C++
> > parser class that can be embedded in my tools.
> >
> > 2) write my own from scratch, maybe first writing a DumbParser class
> > that just provides with a bunch of basic string function, acting like
> > a Lexer and derive the different needed parsers from that, embedding
> > the grammar inside
> > - Pros : full control on the design and the implementation, in
> > comparison to the obfuscated code from Ley/Yacc
> > - Cons : more development time, grammar kind of hardcoded so not
> > easily maintainable, I'm not very experienced in writing text parsers
> > :)
> >
> > I wonder how people handle file parsing in their projects.
> > Thanks for your help.
> >
> > -Marc
>
>
>   Javier Arevalo
>   Pyro Studios
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:
> With Great Power, Comes Great Responsibility
> Learn to use your power at OSDN's High Performance Computing Channel
> http://hpc.devchannel.org/
> _______________________________________________
> Gamedevlists-general mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-general
> Archives:
> http://sourceforge.net/mailarchive/forum.php?forum_id=557
>