sowap-talk — A talk mailing list about SO WAP!

[Sowap-talk] Re: xmerl

[Johan B. <Joh...@mo...>]

Hi,

trying to establish the sowap-talk list as the main forum for xmerl 
discussions, I cross post this (assuming you'r ok with that!)

Short summary:
The OTP team wants to incorporate xmerl into a forthcoming OTP release. 
Bertil Karlsson has volunteered to do some work to make this actually 
happen. Ulf is positive (and so am I in principle), but this also raises 
the question on how xmerl should be maintained regarding the relation 
between sowap-xmerl and OTP-xmerl. In particular, how should future 
changes/extensions be handled. In particular Ulf express his opinion 
that the parser is badly designed and should be rewritten. In particular 
he worries that the parser has a complicated entity expansion, but also 
doesn't always handle character streams.


Now, regarding tests, we already have support for the W3C test suit (see 
xmerl_test.erl) and I think that should be enough for testing the XML 
parser. Needless to say, there are lots of these tests that xmerl 
currently don't handle. To fix these will be a main target for 
xmerl-0.20. Left then are testing of specific configurations of 
xmerl_scan, the export interfaces, the XSL support and other extensions. 
Again, some tests can be found in xmerl_test.erl
A User Guide would be just great to have, see the documentation in 
xmerl-0.19 for at least a start of that (lots can still be done).

I would agree with Ulf that there are issues still with the parser. I'm 
also hoping that these can be fixed in an incremental way. But I would 
suspect there might be backward incompatible changes needed (and I do 
want to have full XML support!). Making backward incompatible changes 
would be much harder in a commercial available OTP-xmerl, but I would 
not like that to be in the way for development on the sowap-xmerl.
Currently there are lots of the W3C tests not handled, new options have 
been added to the parser with every release, and in the latest release 
the SAX export functionality is (somewhat) documented for the first time 
(thus I have probably been the only user...). So, are we happy with the 
stability of xmerl as of today, or should we try to fix some more before 
OTP-xmerl should become a reality?

/Johan

Ulf Wiger (AL/EAB) wrote:
> Hej,
> 
> Personligen skulle jag vilja se det som att xmerl, dvs
> parsern med dess API, exportstödet, m.m. är relativt 
> stabilt och skulle kunna vara del av OTP. Tanken med
> xmerl var ju att den skulle vara enkel att utöka via
> diverse funnar som krokar i valda delar av parsningen.
> 
> Egentligen borde själva parsern skrivas om. Den är 
> feldesignad och kan inte enkelt fixas till att stödja
> hela XML-standarden. Framför allt är det komplicerad
> entity expansion, men också i viss utsträckning 
> character streams, som den inte klarar.
> 
> Sedan borde xmerl nog kompletteras med XML Schema,
> men det är inte något krav jag har för tillfället,
> mest en fundering.
> 
> /Uffe
> 
> -----Original Message-----
> From: Bertil Karlsson [mailto:be...@er...]
> Sent: den 17 november 2003 11:47
> To: Johan Blom; Ulf Wiger (AL/EAB)
> Subject: xmerl
> 
> 
> Hej,
> 
> Kenneth Lundin har önskat xml funktionalitet i OTP. Jag har anmält 
> intresse för saken. Tidigare har jag jobbat med ASN1 applikationen i 
> OTP. Tyvärr har jag inte tidigare erfarenhet av XML, men är intresserad.
> 
> Jag har också förstått att Kenneth har frågat er om det är okej att "ta 
> in" xmerl som en applikation i OTP, och fått ett jakande svar.
> 
> Min plan är att börja med att skriva en testsvit och titta på 
> dokumentationen. Kanske det behövs en "Users Guide".
> 
> Det skulle vara intressant att höra era synpunkter om xmerl i OTP, 
> förhållandet mellan sourceforge projektet och en framtida OTP-xmerl och 
> framtida utökningar av xmerl.
> 
> Vårt huvudsakliga syfte är ju naturligtvis att stödja våra kunders 
> önskemål, men det kan ju ibland sammanfalla med opensource användares.
> 
> 

Re: [Sowap-talk] Re: xmerl, xmerl_scan and state

[<Joh...@mo...>]

Mickael Remond wrote:
> * Joh...@mo... <Joh...@mo...> [2003-11-04 21:34:53 +0100]:
> 
> 
>>Hi Mickael,
>>
>>(I crosspost this to the sowap-talk list, assuming you are ok with that!)
> 
> 
> Of course. No problem !
> 
> 
>>Using the SAX interface (see xmerl_eventp in xmerl-0.19) you would
>>instead export each element when it is parsed, and maintain a user state
>>in the call back module. The result of such a parsing would be what your
>>root element exports.
> 
> 
> I am targeting SAX, indeed. My thought was to use the user_state to
> maintain the result of the parse. This was to avoid creating a state in
> the callback module for each Sax based parse (basically to simplify the
> callback module code).
> 
> My question is thus what is the use of the User_state in a Sax based
> approach ? I could store temporary state or value, but if I maintain a
> state in the call back module, I would rather store the result of the
> parse and the temporary state flags in the callback module state.
> 
> I can understand your position but I do not clearly see what the
> user_state is made for is you are supposed to maintain another state in
> your state parser.
Well, I guess the export interface is something that still needs some 
discussion. For example, currently we have two different ones; one for 
DOM and another one for SAX, which might not be ideal, or?

If have a look at the SAX configurations of the parser in xmerl_eventp 
you will see that I'm indeed using the user_state in the hook function 
to maintain the state. I.e., the user_state is passed as an additional 
parameter to the export callback module, and the selected tag function 
is assumed to return an updated state value. This value is then stored 
in the user state.
This is, however, just one way of doing it and I'm not sure what is the 
optimal solution.

> 
> 
>>>I am using the jungerl one. Maybe it would be nice to maintain the
>>>current version in Jungerl. Whatever are your choice, could you tell me
>>>what is the reference version ?
>>
>>There are actually many reasons why I'm not happy moving to jungerl.
>>Most important perhaps the the close connections with all the other
>>projects hosted at sowap.
> 
> 
> No problem for me, but this should be clearly stated somewhere that the
> reference version is maintained on the sowap CVS.
> I will put a notice in jungerl Xmerl CVS module.
Well, yes but where? At http://www.erlang.org/user.html#xmerl-0.17 there 
is, and has been since ages, a pointer to this site for further updates. 
As this is where Ulf used to publish his versions before, that must be 
the natural place to start looking, or?
But I have indeed been very quiet, this website needs a general overhaul 
etc., so I guess I'm part of the problem.

The jungerl version was added without my knowledge. As anyone is free to
create their own versions of xmerl I don't have any problem with that. 
But of course I would prefer if we could just stick to one cvs repository.


Regards
Johan

[Sowap-talk] Re: xmerl, xmerl_scan and state

[Mickael R. <mic...@er...>]

* Joh...@mo... <Joh...@mo...> [2003-11-04 21:34:5=
3 +0100]:

> Hi Mickael,
>=20
> (I crosspost this to the sowap-talk list, assuming you are ok with that=
!)

Of course. No problem !

> Using the SAX interface (see xmerl_eventp in xmerl-0.19) you would
> instead export each element when it is parsed, and maintain a user stat=
e
> in the call back module. The result of such a parsing would be what you=
r
> root element exports.

I am targeting SAX, indeed. My thought was to use the user_state to
maintain the result of the parse. This was to avoid creating a state in
the callback module for each Sax based parse (basically to simplify the
callback module code).

My question is thus what is the use of the User_state in a Sax based
approach ? I could store temporary state or value, but if I maintain a
state in the call back module, I would rather store the result of the
parse and the temporary state flags in the callback module state.

I can understand your position but I do not clearly see what the
user_state is made for is you are supposed to maintain another state in
your state parser.

> >I am using the jungerl one. Maybe it would be nice to maintain the
> >current version in Jungerl. Whatever are your choice, could you tell m=
e
> >what is the reference version ?
> There are actually many reasons why I'm not happy moving to jungerl.
> Most important perhaps the the close connections with all the other
> projects hosted at sowap.

No problem for me, but this should be clearly stated somewhere that the
reference version is maintained on the sowap CVS.
I will put a notice in jungerl Xmerl CVS module.

Thank you !

--=20
Micka=EBl R=E9mond
 http://www.erlang-projects.org/

[Sowap-talk] Re: xmerl, xmerl_scan and state

[<Joh...@mo...>]

Hi Mickael,

(I crosspost this to the sowap-talk list, assuming you are ok with that!)

Mickael Remond wrote:
> Hello,
> 
> I am not sure that I am using xmerl properly, but I wonder if this would
> not be a good idea to return the last State from xmerl_scan:string and
> file, in addition to Result|Tail ?
> 
> I am doing event based parsing and I am always using the User state to
> "accumulate" the result of the parse. However, there is no easy way to
> get the final user state back to the function that call the
> xmerl_scan functions.
Well, I would argue you are using it in the wrong way.
xmerl has really support for both DOM and SAX style parsing of an XML
document where the default behaviour is DOM style. Using it this way all
parsing is done first, then you might use the export interface for
processing the parse tree and e.g., accumulate a result of the parse
tree (this is how e.g. edoc works). Disadvantages with this method
includes the memory consumption of storing the complete parse tree and
the cost of traversing the parse tree (small I assume).

Using the SAX interface (see xmerl_eventp in xmerl-0.19) you would
instead export each element when it is parsed, and maintain a user state
in the call back module. The result of such a parsing would be what your
root element exports.


> 
> I am planning to do the change myself (small change, see the string/2
> function in xmerl_scan), but I do not know what is the xmerl reference.
As you understand from my above comment I would argue against such a change.
Regarding xmerl reference, this is completely clearly for me the sowap
repository. This has been the case since almost three years when I
accepted maintainership of xmerl from Ulf Wiger. Admittely new releases
has been sparse. Reason beeing this is a low priority project for me and
I'm trying to be really careful when making any changes, and there are
lots of them really since the last release. If someone is not happy with
that, they are of course welcome to create another branch somewhere
else, or even another branch on sowap.

> 
> I am using the jungerl one. Maybe it would be nice to maintain the
> current version in Jungerl. Whatever are your choice, could you tell me
> what is the reference version ?
There are actually many reasons why I'm not happy moving to jungerl.
Most important perhaps the the close connections with all the other
projects hosted at sowap.

> 
> Thank you in advance for your help !
> 
Well, thanks for your interest in xmerl!

Regards
Johan

[Sowap-talk] ANN: xmerl-0.19

[<Joh...@mo...>]

Hi,

I have finally got myself together and merged all received patches,
patched it myself quite a bit and updated the documentation (now using
edoc!!).

As one can expect (almost a year since the last release...) there are
*LOTS* of new stuff in this release. Apart from improved documentation,
we  now have validation support (Sebastian Becuwe), better namespace
handling (Erik Reitsma, Ulf Wiger), XSL:ish support in the Erlang way
(Mikael Karlsson) and further improvements to the export functionality
(Richard Carlsson). Not to mention a large number of bug fixes.
See
http://cvs.sourceforge.net/viewcvs.py/*checkout*/sowap/xmerl/changes.txt?rev=1.29 

for a more complete list of changes.

Also the following incompatibles should be noted:
- Changed the fetch function to return
      Result ::=
        {ok, {file, Filename}, GlobalState'} |
        {ok, {string, String}, GlobalState'} |
        {ok, not_fetched, GlobalState'}

- xmerl:export/[1,2] now takes a list of valid xmerl defined records,
   used to accept only #xmlElement{}

- Renamed xmerl_eventp:file/2 to xmerl_eventp:stream/2

- Moved is_letter/1,is_namechar/1,detect_charset/1,detect_charset/2 from
   xmerl_scan to xmerl_lib

Enjoy!
Johan Blom
Mobile Arts

[Sowap-talk] New xmerl release soon...

[<Joh...@mo...>]

Hi,

I have finally got myself together and merged all received patches, 
patched it myself quite a bit and updated the documentation (now using 
edoc!!). Although trying hard not to, I may have made some mistakes.
In particular the following incompatibles are known:

- Changed the fetch function to return
      Result ::=
        {ok, {file, Filename}, GlobalState'} |
        {ok, {string, String}, GlobalState'}
instead of as currently
      Result ::=
        {ok, GlobalState'} |
        {ok, {file, Filename}, GlobalState'} |
        {ok, {string, String}, GlobalState'}
The reason is that I would like to use it to fetch external resources 
declared as entity or DTD. If I cannot relay on that a Filename or 
String is returned it is
   + impossible to validate, if DTD and the validate option was set
   + impossible to parse, if ENTITY declaration to external resource

- xmerl:export/[1,2] now takes a list of valid xmerl defined records,
   used to accept only #xmlElement{}

- Renamed xmerl_eventp:file/2 to stream/2

- Moved is_letter/1,is_namechar/1,detect_charset/1,detect_charset/2 from
   xmerl_scan to xmerl_lib


See changes.txt for a more complete list of changes.

Please have a look, and I make a more official release within a week or so.


Thanks
Johan Blom
Mobile Arts

[Sowap-talk] Xmerl Contributions: a validation functions[2]

[sebastien b. <seb...@ce...>]

You can see in this mail README and diff which explains the modification.

Regard's,

Sebastien Becuwe

-- 
Sebastien BECUWE
Project Manager

Come and visit Cellicium at the 3GSM World Congress 2003,
18-21 February 2003, Cannes, France,
Palais des Festivals, Hall 3, Booth H36

CELLICIUM
USSD Mobile interactivity for all

Email : seb...@ce...

Tel. : +33 (0) 1 41 98 67 44
Mob. : + 33 (0) 6 82 31 81 22
Fax. : + 33 (0) 1 41 98 67 47
www.cellicium.com
157, rue des Blains - 92220 Bagneux - France

[Sowap-talk] Xmerl Contributions: a validation functions

[sebastien b. <seb...@ce...>]

Good morning,

I have implemented the following changes in xmerl-0.18:
  - addition of XML validation
  - another way to manage ets rules
  - one bug correction
  - detection of standalone field in xml Element

I will send you a diff and detailed explanations in another email.
This is my first Erlang contribution, so if you have any advice or=20
preferences regarding the presentation, please tell me.

Best regards,

S=E9bastien Becuwe

--=20
Sebastien BECUWE
Project Manager

Come and visit Cellicium at the 3GSM World Congress 2003,
18-21 February 2003, Cannes, France,
Palais des Festivals, Hall 3, Booth H36

CELLICIUM
USSD Mobile interactivity for all

Email : seb...@ce...

Tel. : +33 (0) 1 41 98 67 44
Mob. : + 33 (0) 6 82 31 81 22
Fax. : + 33 (0) 1 41 98 67 47
www.cellicium.com
157, rue des Blains - 92220 Bagneux - France

[Sowap-talk] Xmerl Contributions : a validation function

[sebastien b. <seb...@ce...>]

Good morning,

I have implemented the following changes in xmerl-0.18:
   - addition of XML validation
   - another way to manage ets rules
   - one bug correction
   - detection of standalone field in xml Element

I will send you a diff and detailed explanations in another email.
This is my first Erlang contribution, so if you have any advice or=20
preferences regarding the presentation, please tell me.

Best regards,

S=E9bastien Becuwe

--=20
Sebastien BECUWE
Project Manager

Come and visit Cellicium at the 3GSM World Congress 2003,
18-21 February 2003, Cannes, France,
Palais des Festivals, Hall 3, Booth H36

CELLICIUM
USSD Mobile interactivity for all

Email : seb...@ce...

Tel. : +33 (0) 1 41 98 67 44
Mob. : + 33 (0) 6 82 31 81 22
Fax. : + 33 (0) 1 41 98 67 47
www.cellicium.com
157, rue des Blains - 92220 Bagneux - France