Menu
▾
▴
scrollkeeper-devel
|
From: <las...@Su...> - 2001-05-17 09:41:13
|
Hi, I sent out an email a while ago saying that I moved Scrollkeeper to libxml2. I think for the near future at least we might need to consider the migrated version wildly incompatible with previous releases. This is because of the libxml2 library, mainly the SGML parser built into it. There are two problems: 1. The SGML parser is not perfect, there are DocBook tags it doesnt know about. This is easy to fix though, I already sent a couple of patches to Daniel and I will send more. 2. This is more serious, libxml2 expects either UTF-8 input or encoding specified. There are loads of SGML docs out there that are not UTF-8 and while in XML it is possible to specify the encoding in the header, this is not in SGML (AFAIK). So libxml2 expects the encoding to be passed in by the caller. This means we would need to come up with some way of detecting the encoding based on the contents of the SGML file. Both problems flood the screen with error messages when run on a variety of SGML files from various locales. Any ideas? Laszlo |
|
From: Norman W. <nd...@nw...> - 2001-05-17 12:48:17
|
/ László Kovács <las...@Su...> was heard to say:
| 2. This is more serious, libxml2 expects either UTF-8 input or encoding
| specified. There are loads of SGML docs out there that are not UTF-8 and
| while in XML it is possible to specify the encoding in the header, this
| is not in SGML (AFAIK).
You could put
<?sgml-declaration encoding="ISO-8859-1">
Or something at the top of your SGML documents if you wanted to. The
SGML parser wouldn't care.
Be seeing you,
norm
--
Norman Walsh <nd...@nw...> | Success is relative; it is what we make
http://nwalsh.com/ | of the mess we have made of things.--T.
| S. Eliot
|
|
From: John F. <jf...@in...> - 2001-05-17 12:55:58
|
On Thu, May 17, 2001 at 08:47:58AM -0400, Norman Walsh wrote: > / L=E1szl=F3 Kov=E1cs <las...@Su...> was heard to say: > | 2. This is more serious, libxml2 expects either UTF-8 input or encodi= ng > | specified. There are loads of SGML docs out there that are not UTF-8 = and > | while in XML it is possible to specify the encoding in the header, th= is > | is not in SGML (AFAIK). >=20 > You could put >=20 > <?sgml-declaration encoding=3D"ISO-8859-1"> >=20 > Or something at the top of your SGML documents if you wanted to. The > SGML parser wouldn't care. >=20 While something like that is an option, I think our problem is a large body of completed and/or installed docs out there. It would be nice to have some way of detecting the encoding at run time. Is such a thing possible? Cheers, --=20 John Fleck jf...@in... (h) jf...@ab... (w) http://www.abqjournal.com/scitech/ http://www.inkstain.net/fleck/ |
|
From: <las...@Su...> - 2001-05-17 13:53:51
|
> While something like that is an option, I think our problem is a large > body of completed and/or installed docs out there. It would be nice to > have some way of detecting the encoding at run time. Is such a thing > possible? I dont know about any. I asked some localization people here and they said there is no way. Laszlo |
|
From: Dan M. <d-...@uc...> - 2001-05-18 15:50:58
|
On Thu, 17 May 2001, L=E1szl=F3 Kov=E1cs wrote: > 1. The SGML parser is not perfect, there are DocBook tags it doesnt kno= w > about. This is easy to fix though, I already sent a couple of patches t= o > Daniel and I will send more. I don't understand this, but I'm probably just being dumb. What sort of tags does the SGML parser need to know about? Most tags are defined in the DocBook DTD, not in the parser. > 2. This is more serious, libxml2 expects either UTF-8 input or encoding > specified. There are loads of SGML docs out there that are not UTF-8 an= d > while in XML it is possible to specify the encoding in the header, this > is not in SGML (AFAIK). So libxml2 expects the encoding to be passed in > by the caller. This means we would need to come up with some way of > detecting the encoding based on the contents of the SGML file. Did you discuss this with DV? I thought the reason he put the SGML parse= r into libxml2 was to handle these sort of docs and problems. I doubt he expected all SGML docs to have the encoding specified. Dan |
|
From: <las...@Su...> - 2001-05-18 15:57:50
|
Dan Mueth wrote: > = > On Thu, 17 May 2001, L=E1szl=F3 Kov=E1cs wrote: > = > > 1. The SGML parser is not perfect, there are DocBook tags it doesnt k= now > > about. This is easy to fix though, I already sent a couple of patches= to > > Daniel and I will send more. > = > I don't understand this, but I'm probably just being dumb. What sort o= f > tags does the SGML parser need to know about? Most tags are defined in= > the DocBook DTD, not in the parser. That is a question for DV, not for me. In the source file of the SGML parser there is a big table with the DocBook tags. If the tag is not there then it is not recognized by the parser. > Did you discuss this with DV? I thought the reason he put the SGML par= ser > into libxml2 was to handle these sort of docs and problems. I doubt he= > expected all SGML docs to have the encoding specified. No, he didnt expect them to have the encoding inside them, but the SGML parser routines take an extra parameter which is the encoding. As far as the SGML parser is concerned the encoding has to be passed in by the caller. As far as I see the situation the SGML parser is not able to detect the encoding. I think (seeing Norman's mail) that DocBook SGML does not have any way to specify the encoding in the file while XML in general specifies it in the header. This is why it can be detected in XML, it is not based on the content. Laszlo |
|
From: Daniel V. <vei...@re...> - 2001-05-18 16:07:40
|
On Fri, May 18, 2001 at 04:57:31PM +0100, L=E1szl=F3 Kov=E1cs wrote: > Dan Mueth wrote: > >=20 > > On Thu, 17 May 2001, L=E1szl=F3 Kov=E1cs wrote: > >=20 > > > 1. The SGML parser is not perfect, there are DocBook tags it doesnt= know > > > about. This is easy to fix though, I already sent a couple of patch= es to > > > Daniel and I will send more. > >=20 > > I don't understand this, but I'm probably just being dumb. What sort= of > > tags does the SGML parser need to know about? Most tags are defined = in > > the DocBook DTD, not in the parser. > That is a question for DV, not for me. In the source file of the SGML > parser there is a big table with the DocBook tags. If the tag is not > there then it is not recognized by the parser. libxml don't read SGML Dtds, this parser is an had-hoc parser. All tags need to be compiled in. Sorry it's gross but developping a full blown SGML parser is not a good use of anybody's time now :-) > > Did you discuss this with DV? I thought the reason he put the SGML p= arser > > into libxml2 was to handle these sort of docs and problems. I doubt = he > > expected all SGML docs to have the encoding specified. > No, he didnt expect them to have the encoding inside them, but the SGML > parser routines take an extra parameter which is the encoding. As far a= s > the SGML parser is concerned the encoding has to be passed in by the > caller. As far as I see the situation the SGML parser is not able to > detect the encoding. I think (seeing Norman's mail) that DocBook SGML > does not have any way to specify the encoding in the file while XML in > general specifies it in the header. This is why it can be detected in > XML, it is not based on the content. Yes that's the problem. We need to know what is the encoding used by the SGML document. I don't k= now how to guess/compute it. Daniel --=20 Daniel Veillard | Red Hat Network http://redhat.com/products/network= / vei...@re... | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ |
|
From: Norman W. <nd...@nw...> - 2001-05-18 16:53:00
|
/ Daniel Veillard <vei...@re...> was heard to say:
| libxml don't read SGML Dtds, this parser is an had-hoc parser. All tags
| need to be compiled in. Sorry it's gross but developping a full blown
| SGML parser is not a good use of anybody's time now :-)
And simply linking to SP isn't an option because...?
Be seeing you,
norm
--
Norman Walsh <nd...@nw...> | 'Heartless Cynics,' the young men
http://nwalsh.com/ | shout, / Blind to the world of Fact
| without; / 'Silly Dreamers,' the old
| men grin / Deaf to the world of Purpose
| within.--W. H. Auden
|
|
From: Daniel V. <vei...@re...> - 2001-05-18 17:03:45
|
On Fri, May 18, 2001 at 12:52:53PM -0400, Norman Walsh wrote: > / Daniel Veillard <vei...@re...> was heard to say: > | libxml don't read SGML Dtds, this parser is an had-hoc parser. All tags > | need to be compiled in. Sorry it's gross but developping a full blown > | SGML parser is not a good use of anybody's time now :-) > > And simply linking to SP isn't an option because...? Because I don't know how to do it. If scrollkeeper developers want to do it, fine by me :-) There is no good reason to do it for libxml, the goal of the exercise was to provide a tool (a hack agreed) to smooth the transition (and provide a way to save external parsed entities when converting existing docs to XML). Now if someone has a better solution, fine by me. Daniel -- Daniel Veillard | Red Hat Network http://redhat.com/products/network/ vei...@re... | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ |
|
From: Gregory L. <gle...@cu...> - 2001-05-18 16:05:23
|
On 18 May 2001 10:44:54 -0500, Dan Mueth wrote:
>
> On Thu, 17 May 2001, László Kovács wrote:
>
> > 1. The SGML parser is not perfect, there are DocBook tags it doesnt know
> > about. This is easy to fix though, I already sent a couple of patches to
> > Daniel and I will send more.
>
> I don't understand this, but I'm probably just being dumb. What sort of
> tags does the SGML parser need to know about? Most tags are defined in
> the DocBook DTD, not in the parser.
I think your confusion comes from calling this a parser. It's not a
fully-fledged SGML parser, just a "hack" to allow us to process out SGML
docs. I'm still a bit torn here, but I think we might want to look at
taking advantage of the code in jade or openjade to do a real SGML to
XML conversion for us, on the fly (since it seems to only take a
fraction of a second for the conversion using sx/osx). I don't know how
unfriendly that would be.
Greg
--
Troll, troll, troll your post
Gently down the feed
Merrily, merrily troll along
A life is what you need...
|
|
From: Daniel V. <vei...@re...> - 2001-05-18 16:10:14
|
On Fri, May 18, 2001 at 09:04:26AM -0700, Gregory Leblanc wrote: > I think your confusion comes from calling this a parser. It's not a > fully-fledged SGML parser, just a "hack" to allow us to process out SGML > docs. I'm still a bit torn here, but I think we might want to look at > taking advantage of the code in jade or openjade to do a real SGML to > XML conversion for us, on the fly (since it seems to only take a > fraction of a second for the conversion using sx/osx). I don't know how > unfriendly that would be. Sure you can use the existing tools to convert your content to XML. But all our docs uses external parsed entities. Your scheme won't deal with this cleanly (basically you will get everything substitued in a single flat file as the output, I doubt anybody who wrote docs for Gnome will like this). Now if you feel compelled for providing a better solution I would be *extremely* pleased if you could volunteer to do this conversion for the GNOME project in general, thanks you in advance ! Daniel -- Daniel Veillard | Red Hat Network http://redhat.com/products/network/ vei...@re... | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ |
|
From: <las...@Su...> - 2001-05-21 12:46:20
|
Gregory Leblanc wrote: > = > On 18 May 2001 10:44:54 -0500, Dan Mueth wrote: > > > > On Thu, 17 May 2001, L=E1szl=F3 Kov=E1cs wrote: > > > > > 1. The SGML parser is not perfect, there are DocBook tags it doesnt= know > > > about. This is easy to fix though, I already sent a couple of patch= es to > > > Daniel and I will send more. > > > > I don't understand this, but I'm probably just being dumb. What sort= of > > tags does the SGML parser need to know about? Most tags are defined = in > > the DocBook DTD, not in the parser. > = > I think your confusion comes from calling this a parser. It's not a > fully-fledged SGML parser, just a "hack" to allow us to process out SGM= L > docs. I'm still a bit torn here, but I think we might want to look at > taking advantage of the code in jade or openjade to do a real SGML to > XML conversion for us, on the fly (since it seems to only take a > fraction of a second for the conversion using sx/osx). I don't know ho= w > unfriendly that would be. > Greg > = I have to rectify this, the problems are not because of the parser, but because of the poor quality of the DocBook/SGML docs I am parsing. This is the latest Ximian Gnome docs, some of them having embarassing errors in them. Laszlo |
|
From: <las...@su...> - 2001-05-18 17:25:05
|
> And simply linking to SP isn't an option because...? > > Be seeing you, > norm What is SP? Laszlo |
|
From: Norman W. <Nor...@ea...> - 2001-05-18 18:39:15
|
/ László Kovács <las...@su...> was heard to say:
| > And simply linking to SP isn't an option because...?
|
| What is SP?
James Clark's SGML parser. The heart of nsgmls, jade, sx, etc.
Be seeing you,
norm
--
Nor...@Su... | Everything should be made as simple as
XML Standards Engineer | possible, but no simpler.
Technology Dev. Group |
Sun Microsystems, Inc. |
|
|
From: Dan M. <d-...@uc...> - 2001-05-18 19:20:25
|
On 18 May 2001, Norman Walsh wrote: > / L=E1szl=F3 Kov=E1cs <las...@su...> was heard to say: > | > And simply linking to SP isn't an option because...? > | > | What is SP? > > James Clark's SGML parser. The heart of nsgmls, jade, sx, etc. How do these deal with the lack of encoding specification for DocBook/SGM= L docs? It isn't obvious to me why it is a problem for XML docs and libxml= 2 but not for SGML docs with jade/SP. Dan |
|
From: Norman W. <Nor...@ea...> - 2001-05-18 19:48:10
|
/ Dan Mueth <d-...@uc...> was heard to say:
| How do these deal with the lack of encoding specification for DocBook/SGML
| docs? It isn't obvious to me why it is a problem for XML docs and libxml2
| but not for SGML docs with jade/SP.
I'm not sure.
Be seeing you,
norm
--
Nor...@Su... | The truest wild beasts live in the most
XML Standards Engineer | populous places.--Graci\'an
Technology Dev. Group |
Sun Microsystems, Inc. |
|
|
From: Daniel V. <vei...@re...> - 2001-05-19 06:57:37
|
On Fri, May 18, 2001 at 03:48:05PM -0400, Norman Walsh wrote: > / Dan Mueth <d-...@uc...> was heard to say: > | How do these deal with the lack of encoding specification for DocBook/SGML > | docs? It isn't obvious to me why it is a problem for XML docs and libxml2 > | but not for SGML docs with jade/SP. > > I'm not sure. Argh, if you don't know, who will ... Maybe we will have to ask James. BTW, if automatic encoding detection is not possible, is there a way in DocBook to indicate the encoding if known (for example on the root element) ? If this detection/advertizing requires human input, it would be great to have to deal with it only once. Daniel -- Daniel Veillard | Red Hat Network http://redhat.com/products/network/ vei...@re... | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ |
|
From: Norman W. <Nor...@ea...> - 2001-05-19 16:47:58
|
/ Daniel Veillard <vei...@re...> was heard to say:
| On Fri, May 18, 2001 at 03:48:05PM -0400, Norman Walsh wrote:
| > / Dan Mueth <d-...@uc...> was heard to say:
| > | How do these deal with the lack of encoding specification for DocBook/SGML
| > | docs? It isn't obvious to me why it is a problem for XML docs and libxml2
| > | but not for SGML docs with jade/SP.
| >
| > I'm not sure.
|
| Argh, if you don't know, who will ... Maybe we will have to ask James.
The answer occurred to me while I was working in the garden this
morning. While XML has a fixed SGML Declaration, SGML does not. So if
you want to use some character encoding other than the one described
by the reference concrete syntax, you have to provide the SGML
Declaration that indicates what characters are used (and what they
mean).
Although now that I write this, I'm not sure if that's the whole answer.
Just because I say that code point 0xED is a name character that doesn't
tell the processor what glyph to use. Hmm...
Be seeing you,
norm
--
Nor...@Su... | Resist the urge to hurry; it will only slow
XML Standards Engineer | you down--Bruce Eckel
Technology Dev. Group |
Sun Microsystems, Inc. |
|
|
From: Eric B. <e.b...@no...> - 2001-05-21 15:41:00
|
Le Saturday 19 May 2001 08:57, Daniel Veillard a =E9crit : > On Fri, May 18, 2001 at 03:48:05PM -0400, Norman Walsh wrote: > > / Dan Mueth <d-...@uc...> was heard to say: > > | How do these deal with the lack of encoding specification for > > | DocBook/SGML docs? It isn't obvious to me why it is a problem for = XML > > | docs and libxml2 but not for SGML docs with jade/SP. > > > > I'm not sure. > > Argh, if you don't know, who will ... Maybe we will have to ask James. I know probably as much, or less, as Norm. Here is what I know SP (jade, SX, etc) has defined rules about the encoding to use. These rul= es=20 are based on the document's contents as well as the system's default=20 character set and environment variables. The detailed rules are on: http://www.jclark.com/sp/charset.htm > BTW, if automatic encoding detection is not possible, is there a way in > DocBook to indicate the encoding if known (for example on the root elem= ent) > ? If this detection/advertizing requires human input, it would be great= to > have to deal with it only once. --=20 +----------------------------------------------------------------------+ | =C9ric Bischoff mailto:e.b...@no... = | +----------------------------------------------------------------------+ |
|
From: Eric B. <e.b...@no...> - 2001-05-21 15:23:43
|
Le Friday 18 May 2001 21:48, Norman Walsh a =E9crit : > / Dan Mueth <d-...@uc...> was heard to say: > | How do these deal with the lack of encoding specification for > | DocBook/SGML docs? It isn't obvious to me why it is a problem for XM= L > | docs and libxml2 but not for SGML docs with jade/SP. > > I'm not sure. At KDE we assumed that <book lang=3D"ru"> induced koi-8r encoding, <book=20 lang=3D"fr"> induced iso-8859-15 encoding, etc. (now we moved to XML and UTF-8 so this is history) --=20 +----------------------------------------------------------------------+ | =C9ric Bischoff mailto:e.b...@no... = | +----------------------------------------------------------------------+ |
|
From: <las...@su...> - 2001-05-21 16:04:27
|
> At KDE we assumed that <book lang="ru"> induced koi-8r encoding, <book > lang="fr"> induced iso-8859-15 encoding, etc. > > (now we moved to XML and UTF-8 so this is history) Happy people:-) I wonder if it could be made history for Gnome too. Laszlo |
|
From: Karl E. <ke...@gm...> - 2001-05-22 05:31:57
|
L=E1szl=F3 Kov=E1cs <las...@su...> writes:
> > (now we moved to XML and UTF-8 so this is history)
> Happy people:-) I wonder if it could be made history for Gnome too.=20
No need to "move" -- just delegate:
Don't install SGML files but "convert" them at installation time to
XML (sgmlnorm and s2x can do wonders). Proposed earlier by me.
It's still beyond me why the doc team insists on "We Want Our
'Tools' to Work on 'pristine' SGML files".
Don't waste your time writing hacks to process SGML files for
display purposes. These days SGML is for writers only (more or
less).
--=20
work : ke...@su... | ,__o
: http://www.suse.de/~ke/ | _-\_<,
home : ke...@gm... | (*)/'(*)
|
|
From: Dan M. <d-...@uc...> - 2001-05-23 17:53:05
|
On 22 May 2001, Karl Eichwalder wrote: > L=E1szl=F3 Kov=E1cs <las...@su...> writes: > > > > (now we moved to XML and UTF-8 so this is history) > > Happy people:-) I wonder if it could be made history for Gnome too. > > No need to "move" -- just delegate: > > Don't install SGML files but "convert" them at installation time to > XML (sgmlnorm and s2x can do wonders). Proposed earlier by me. Karl - Thanks for the suggestion :) This may be a really nice way to circumvent the encoding issues and support SGML documents without much work. Does anybody know exactly what happens w/r/t encodings when we do something like: sgml2xml -b UTF-8 foo.sgml > foo.xml It looks like this uses the encoding detection in SP to convert from an arbitrary unknown encoding to UTF? Does anybody see any problems with using a call like this in ScrollKeeper to convert any SGML docs to XML for further processing by libxml2/xslt in ScrollKeeper? This way we can keep our SGML support without having to do much work or add bloat to ScrollKeeper or libxml2. Dan |
|
From: Karl E. <ke...@gm...> - 2001-05-24 09:34:44
|
Dan Mueth <d-...@uc...> writes:
> Karl - Thanks for the suggestion :)
I'm always at yous disposal ;)
> This may be a really nice way to circumvent the encoding issues and
> support SGML documents without much work.
>
> Does anybody know exactly what happens w/r/t encodings when we do
> something like:
> sgml2xml -b UTF-8 foo.sgml > foo.xml
Sorry, don't know (it works for iso-8859-1).
> It looks like this uses the encoding detection in SP to convert from an
> arbitrary unknown encoding to UTF?
Maybe, proprocessing with recode/iconv is needed. The writer should
store the encoding somewhere (inside of a PI or a comment).
> Does anybody see any problems with using a call like this in ScrollKeeper
> to convert any SGML docs to XML for further processing by libxml2/xslt in
> ScrollKeeper?
No :)
--
work : ke...@su... | ,__o
: http://www.suse.de/~ke/ | _-\_<,
home : ke...@gm... | (*)/'(*)
|
|
From: <las...@Su...> - 2001-05-24 14:06:43
|
Dan Mueth wrote: > = > On 22 May 2001, Karl Eichwalder wrote: > = > > L=E1szl=F3 Kov=E1cs <las...@su...> writes: > > > > > > (now we moved to XML and UTF-8 so this is history) > > > Happy people:-) I wonder if it could be made history for Gnome too.= > > > > No need to "move" -- just delegate: > > > > Don't install SGML files but "convert" them at installation time = to > > XML (sgmlnorm and s2x can do wonders). Proposed earlier by me. > = > Karl - Thanks for the suggestion :) > = > This may be a really nice way to circumvent the encoding issues and > support SGML documents without much work. > = > Does anybody know exactly what happens w/r/t encodings when we do > something like: > sgml2xml -b UTF-8 foo.sgml > foo.xml > = > It looks like this uses the encoding detection in SP to convert from an= > arbitrary unknown encoding to UTF? > = > Does anybody see any problems with using a call like this in ScrollKeep= er > to convert any SGML docs to XML for further processing by libxml2/xslt = in > ScrollKeeper? > = > This way we can keep our SGML support without having to do much work or= > add bloat to ScrollKeeper or libxml2. > = > Dan I tried to build OpenSP on both Solaris and Linux and I failed because of some C++ related problem coming from the source. = On the other hand sgml2xml seemed extremely poorly documented, I couldnt really figure out how to use it. Maybe it needs more looking. If the SGML encoding issue takes a long time to be sorted, we might want to consider separate releases for SGML and XML i18n support. The i18n improvements work for XML that I worked on was finished a couple of days ago and the SGML encoding issue is the only remaining problem right now. Laszlo |