scrollkeeper-devel

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Wed, 18 Oct 2000, Eric Bischoff wrote:

> laszlo kovacs wrote:
> > 
> > Hi Eric,
> > 
> > Thanks for this, but I don't know much about packaging in general so I
> > don't have much clue about what you are suggesting.
> 
> Let me explain you.
> 
> In Linux world, it's not the same person that programs the software and
> that packages the software, unlike in Windows world.
> 
> The software programmer produces a Makefile. This makefile has an
> "install" section that is enough to install the software locally.
> 
> The packager takes this makefile, runs it on his own computer, and sees
> what comes of it. He usually gets a set of files that will go into some
> list and be distributed.
> 
> The problem appears when the makefile does not simply create a
> standalone file, but *modifies* a centralized database. That's exactly
> what will happen with scrollkeeper-install. There's no resulting file
> that can simply be thrown into the distribution: the result is a
> *portion* of a file. So the packager puts the corresponding instructions
> in what is called a "post-installation" script. The packager builds the
> package on his own computer, but he knows the post-installation script
> will be run on the end-user's computer.
> 
> This is why Dan's setup would simply not have worked.
> 
> The most efficient solution and that will not require too much changes
> in existing makefiles and specfiles is to split the script in two:
> - one part, called in "install" section of the makefile, gathers all the
> OMF info into a "metainformation snippet"
> - another part, called in the post-installation script of the specfile
> (or whatever equivalent), merges this snippet with the existing
> database, which holds both the contents list and the OMF information.
> 
> > Anyway the install
> > and uninstall scripts are very easy to change as they are small so any
> > change can be done quickly.
> 
> Better so. Here is in short what I propose.
> 
> Before:
> > --------------------------------
> > In the Makefile:
> > > install:
> > >         cp foo.sgml $(DOCDIR)
> > >         cp foo-omf $(DOCDIR)
> > >         scrollkeeper-install $(DOCDIR)/foo.sgml $(DOCDIR)/foo-omf
> > --------------------------------
> 3 lines *per document* in the Makefile
> 
> After:
> > --------------------------------
> > In the Makefile:
> > > install:
> > >         cp foo.sgml $(DOCDIR)
> > >         scrollkeeper-preinstall $(DOCDIR)/foo.sgml foo.omf > $(DOCDIR)/doc-contents.omf
> >
> > In the SPEC file:
> > > %Post
> > >         scrollkeeper-install $DOCDIR/doc-contents.omf
> > >
> > > %Postun
> > >         scrollkeeper-uninstall $DOCDIR/doc-contents.omf
> > >
> > > %Files
> > >	    $DOCDIR/doc-contents.omf
> > --------------------------------
> 2 lines *per document* in the makefile
> 3 lines *for all documents* in the specfile


Unfortunately - when people './configure;make;make install' they will 
miss out on this aspect.

What you might need to do is to create some sort of 
system/cron-job/daemon in which scrollkeeper checks the local files 
installed - if it is missing it will remove them from its internal 
database.

Otherwise you will need a big huge README in each project saying "please 
run scrollkeeper-install after 'make install' is done"

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Wed, 18 Oct 2000, laszlo kovacs wrote:

> > Unfortunately - when people './configure;make;make install' they will
> > miss out on this aspect.
> > 
> > What you might need to do is to create some sort of
> > system/cron-job/daemon in which scrollkeeper checks the local files
> > installed - if it is missing it will remove them from its internal
> > database.
> > 
> > Otherwise you will need a big huge README in each project saying "please
> > run scrollkeeper-install after 'make install' is done"
> 
> Can each project have in its Makefile a section that checks for
> Scrollkeeper and if it's installed then installs the docs for
> Scrollkeeper also (same for uninstall)? I guess the problem is the large
> number of packages and distributors, but it can be done in time if the
> system (Scrollkeeper and the Help Browser) is good so distributors see
> it is better to support it.

The problem arised when you are building RPMs (for example).

You do an 'rpm -ta file.tar.gz' - It starts building and then does a 
'make install' to detect all the installed files. Unfortunately during 
this step you also do the 'scrollkeeper-install' thingy (so the file gets 
installed with scrollkeeper even though you were just BUILDING a package 
and not INSTALLING a package). 

Icky situation, right?
 
> Laszlo
> 

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> The problem arised when you are building RPMs (for example).
> 
> You do an 'rpm -ta file.tar.gz' - It starts building and then does a
> 'make install' to detect all the installed files. Unfortunately during
> this step you also do the 'scrollkeeper-install' thingy (so the file gets
> installed with scrollkeeper even though you were just BUILDING a package
> and not INSTALLING a package).
> 
> Icky situation, right?

As you say.

I feel less preaching in the desert now you described the problem in so
concise and understandable terms! Thanks for the contribution, Ali.

-- 
 Éric Bischoff   -   mailto:eb...@cy...
 __________________________________________________
                                           \^o~_.
     .~.                           ______  /( __ )
     /V\         Toys story         \__  \/  (  V
   //   \\                            \__| (__=v
  /(     )\                        |\___/     )
    ^^-^^                           \_____(  )
     Tux                        Konqui     \__=v
 __________________________________________________

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Wed, 18 Oct 2000, Dan Mueth wrote:

> 
> On Wed, 18 Oct 2000, Eric Bischoff wrote:
> 
> > After:
> > > --------------------------------
> > > In the Makefile:
> > > > install:
> > > >         cp foo.sgml $(DOCDIR)
> > > >         scrollkeeper-preinstall $(DOCDIR)/foo.sgml foo.omf > $(DOCDIR)/doc-contents.omf
> > >
> > > In the SPEC file:
> > > > %Post
> > > >         scrollkeeper-install $DOCDIR/doc-contents.omf
> > > >
> > > > %Postun
> > > >         scrollkeeper-uninstall $DOCDIR/doc-contents.omf
> > > >
> > > > %Files
> > > >	    $DOCDIR/doc-contents.omf
> > > --------------------------------
> 
> Thanks Eric.  It is very nice to get input from somebody who knows a lot
> about packaging.
> 
> I generally like this approach.  I have a few concerns:
> 
> 1) Say we have foo/app1/docs/app1_doc1 and foo/app2/docs/app2_doc1.  Then
> scrollkeeper-preinstall will overwrite $(DOCDIR)/doc-contents.omf twice
> and the final would only have the last doc listed. We could use ">>", but
> if $(DOCDIR)/doc-contents.omf happens to pre-exist, then we keep whatever
> crud was in it from before.
> 
> 2) As Ali pointed out, it seems to be hard to "have our cake and eat it
> too". Either 'make install' actually calls scrollkeeper-install on the
> local machine (as needed by tarball installs), or else it doesn't (as
> people who want to merely create an RPM but not install it would prefer).  
> The solution eludes me, although I'm sure this has been encountered many
> times in the past and there must be some nice solution.  A not too nice
> solution would be to have a separate target that the spec file calls
> instead of 'install'.  So we'd have 'make install' and 'make
> installforrpm'.  This seems inelegant though.  Any RPM experts out there
> who would like to point out the solution?  Eric?  Also, I am trusting that
> deb's will not pose any new problems here.

There is a solution to this. Include a patch to the RPM that will 
"modify" the tarball to NOT run scrollkeeper-install (patches can be 
"attached" to RPMS (RedHat does it, but I do not know the technical side 
of how it is done)). This is the only "sound" solution I can think of. 
Debian dudes have their own patches to handle stuff :) 

Unfortunately, you have to ask yourself - Will application writers want 
to do this for their packages?
 
> 3) Do deb's let you control where the deb installs?  The way we are doing
> things now, we are writing the path of the SGML file into the OMF metadata
> in the pre-install script. So if the installer has any power to change
> where things get installed (say under /share instead of
> /usr/local/share) then everything breaks.
> 
> 4) Did you literally mean to use the name "doc-contents.omf"?  This makes
> me think of the contents list which isn't really what this file is.  If I
> understand you (Eric), this is basically just a file with all the OMF
> files concatenated and possibly a bit more info if necessary.  Also, since
> there is a possibility of multiple packages installing docs into the same
> directory, we probably want to give them unique names. Perhaps
> $(DOCDIR)/<packagename>-docs.omf?
> 
> Dan
> 
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel
> 

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> There is a solution to this. Include a patch to the RPM that will
> "modify" the tarball to NOT run scrollkeeper-install (patches can be
> "attached" to RPMS (RedHat does it, but I do not know the technical side
> of how it is done)). This is the only "sound" solution I can think of.
> Debian dudes have their own patches to handle stuff :)

It's no clean solution and it can be tedious to prepare patches.

No the better is to let "make install" behave normally and use somehow
the partial database it built, either by ensuring it is copied in some
common directory but with a different name as the other database
snippets and that the help browser copes with a database spread over
many files, or that it gets merged together with the "real" database on
the user's hard disk. The first solution is the one that requires less
work for the distribution packager.

> Unfortunately, you have to ask yourself - Will application writers want
> to do this for their packages?

-- 
 Éric Bischoff   -   mailto:eb...@cy...
 __________________________________________________
                                           \^o~_.
     .~.                           ______  /( __ )
     /V\         Toys story         \__  \/  (  V
   //   \\                            \__| (__=v
  /(     )\                        |\___/     )
    ^^-^^                           \_____(  )
     Tux                        Konqui     \__=v
 __________________________________________________

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Mon, 23 Oct 2000, laszlo kovacs wrote:

> [snip]
> 
> > The line "scrollkeeper somearguments" in the "install:" section of the
> > Makefile adds or remove information in some kind of database, right?
> > Some kind of pointer to the real help file and to the OMF file,
> > information about the category in the contents list it belongs too, or
> > some other kind of information? *THIS IS THE PART THAT MIGHT CAUSE A
> > PROBLEM*. When the packager runs "make install", then either:
> > 
> > a) Scrollkeeper assumes that the database exists previously on the hard
> > disk - then make install will simply fail, because packages are built
> > usually on /tmp with what is called "buildrooting". This means that if
> > you database is normally in /usr/share/scrollkeeper, then make install
> > works with /tmp/usr/share/scrollkeeper. (BTW, this means that
> > scrollkeeper should accept a --prefix argument or something similar).
> Scrollkeeper uses the prefix argument. It knows where it was installed
> and in the current implementation all the files created or updated by it
> (including the database) go into prefix/var/lib/scrollkeeper. So
> according to the current implementation scrollkeeper will always find
> from a  "make install"  the right database even if that is run from the
> command line or from an RPM creation process. I just realised this so
> Dan's suggestion wouldn't work (at least according to the current
> implementation).

Would it be possible to change the database using an environment 
variable? ;)

Secondly, how about .spec files set an environment variable to 
OMF_NO_INSTALL when building? (and unset it when done) - Then 
scrollkeeper can detect it and just not install the file into its 
database.

This would work for tarball users too :) 

Sure, there are corner cases where it won't work (compiling an RPM in the 
background and compiling a tarball in the foreground (with rougly 
similiar install times)). But I don't like corner cases :P

> > b) If the database does not exist previously, then it creates it. Then
> > in /tmp you will find a partial database with the information relative
> > only to the docs that are in this package. Either this partial database
> > can be simply copied under a unique name and the help browser must be
> > able to cope with a database that is split over several files, either
> > there's a "merge" option to scrollkeeper that must be copied into the
> > postinstallation section of the spec file (or install.sh on a
> > slackware), and the distribution packager has four extra lines to add to
> > his spec file and he curses you.
> Multiple database files processing in the Help Browser could be
> implemented obviously. The problem here is that Scrollkeeper is supposed
> to be the universal document cataloging system for every help browser.
> That means it is supposed to offer the simplest possible interface for
> the browser. However the reality at the moment is that the only help
> browser that will use Scrollkeeper in the near future is Nautilus and we
> develop the Nautilus module that will use Scrollkeeper so we can adapt
> the code easily. One possible solution would be that we would supply a
> library that would compile the partial database files to one XML DOM
> tree and pass that to the browser. I wonder how much overhead this would
> create at start-up if there are a couple of hundreds (or thousands) of
> partial database files.

I don't think you should pass the XML DOM tree. I think there should be 
API functions (or commands or whatever) to allow you to get every item in 
the tree. Also, to speed things up, there should be a cache (so you don't 
have to re-build the XML DOM tree all the time).

Regards,
Ali

> Laszlo
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel
> 

Re: [Scrollkeeper-devel] What about the distributions?

[laszlo k. <las...@su...>]

> Would it be possible to change the database using an environment
> variable? ;)
> 
> Secondly, how about .spec files set an environment variable to
> OMF_NO_INSTALL when building? (and unset it when done) - Then
> scrollkeeper can detect it and just not install the file into its
> database.
> 
> This would work for tarball users too :)
> 
> Sure, there are corner cases where it won't work (compiling an RPM in the
> background and compiling a tarball in the foreground (with rougly
> similiar install times)). But I don't like corner cases :P
I dont know if this breaks Eric's principle of doing the same thing by
make install when is run for package creation or when it is run for
install. For Solaris we would still need post install scripts. I would
go for either post install scripts or partial databases, but we should
keep it simple, so I dont like the idea of using different approaches
for different package formats. 

> I don't think you should pass the XML DOM tree. I think there should be
> API functions (or commands or whatever) to allow you to get every item in
> the tree. Also, to speed things up, there should be a cache (so you don't
> have to re-build the XML DOM tree all the time).
Maybe. I was trying to stress the importance of presenting a unified
database rather than lots of partial databases to the help browser (in
case Scrollkeeper output would be partial databases). The implementation
is not that important at the moment.

Laszlo

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

laszlo kovacs wrote:
> 
> I dont know if this breaks Eric's principle of doing the same thing by
> make install when is run for package creation or when it is run for
> install.

It's not *my* principle ;-).

> For Solaris we would still need post install scripts.

Sure, on every system you would need them, should it be RPM, DEB,
slackware or anything else.

You would need them, unless the browser is able to work with partial
databases.

> I would
> go for either post install scripts or partial databases, but we should
> keep it simple, so I dont like the idea of using different approaches
> for different package formats.

Who speaks about that?

The problem with any distribution (Solaris or other) is that "make
install" is not run on end-user's machine. Therefore either you merge
the partial database that you obtain after "make install" into a bigger
single database during the post-installation, either you simply copy
this partial database during installation and hope that it will work
with many small databases instead of a big one.


-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> Secondly, how about .spec files set an environment variable to
> OMF_NO_INSTALL when building? (and unset it when done) - Then
> scrollkeeper can detect it and just not install the file into its
> database.

Much simpler, if you want to throw away this information, just make
scrollkeeper
do nothing if the database does not exist previously.

But I think it's a better idea to keep this partial information and have
it either
merged into the user's database or directly used by the web browser.

> > Multiple database files processing in the Help Browser could be
> > implemented obviously. The problem here is that Scrollkeeper is supposed
> > to be the universal document cataloging system for every help browser.
> > That means it is supposed to offer the simplest possible interface for
> > the browser. However the reality at the moment is that the only help
> > browser that will use Scrollkeeper in the near future is Nautilus and we
> > develop the Nautilus module that will use Scrollkeeper so we can adapt
> > the code easily. One possible solution would be that we would supply a
> > library that would compile the partial database files to one XML DOM
> > tree and pass that to the browser. I wonder how much overhead this would
> > create at start-up if there are a couple of hundreds (or thousands) of
> > partial database files.

If we adopt such a solution, there has to be a caching mechanism. You
can't
rebuild it all at the startup every time.

> I don't think you should pass the XML DOM tree. I think there should be
> API functions (or commands or whatever) to allow you to get every item in
> the tree. Also, to speed things up, there should be a cache (so you don't
> have to re-build the XML DOM tree all the time).

Yes, if the browser can cope with partial information, and pick up in
several
files when it needs some more information, then the problem is solved.


-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ali...@au...>]

* Eric Bischoff (eb...@cy...) wrote at 23:25 on 25/10/00:
> Ali Abdin wrote:
> > 
> > Secondly, how about .spec files set an environment variable to
> > OMF_NO_INSTALL when building? (and unset it when done) - Then
> > scrollkeeper can detect it and just not install the file into its
> > database.
> 
> Much simpler, if you want to throw away this information, just make
> scrollkeeper
> do nothing if the database does not exist previously.
> 
> But I think it's a better idea to keep this partial information and have
> it either
> merged into the user's database or directly used by the web browser.

I would like to point out that most packages do not add '--localstatedir=/var'
in their spec files :P

So you may end up with a scrollkeeper DB in '/usr/var' and '/var' :P
 
> > > Multiple database files processing in the Help Browser could be
> > > implemented obviously. The problem here is that Scrollkeeper is supposed
> > > to be the universal document cataloging system for every help browser.
> > > That means it is supposed to offer the simplest possible interface for
> > > the browser. However the reality at the moment is that the only help
> > > browser that will use Scrollkeeper in the near future is Nautilus and we
> > > develop the Nautilus module that will use Scrollkeeper so we can adapt
> > > the code easily. One possible solution would be that we would supply a
> > > library that would compile the partial database files to one XML DOM
> > > tree and pass that to the browser. I wonder how much overhead this would
> > > create at start-up if there are a couple of hundreds (or thousands) of
> > > partial database files.
> 
> If we adopt such a solution, there has to be a caching mechanism. You
> can't
> rebuild it all at the startup every time.

By the way, I'm beginning to think that we may want to expose the xmlDocPtr
(or xmlNodePtr) (of an OMF file) to the application. That way application
authors will have more "control" over the data, and they can implement their
own caching mechanisms if they wish.
 
> > I don't think you should pass the XML DOM tree. I think there should be
> > API functions (or commands or whatever) to allow you to get every item in
> > the tree. Also, to speed things up, there should be a cache (so you don't
> > have to re-build the XML DOM tree all the time).
> 
> Yes, if the browser can cope with partial information, and pick up in
> several
> files when it needs some more information, then the problem is solved.
> 
> 
> -- 
> Éric Bischoff  -  Documentation and Localization
> Caldera (Deutschland) GmbH - Linux for eBusiness
> Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
> http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

> I would like to point out that most packages do not add
> '--localstatedir=/var' in their spec files :P
>
> So you may end up with a scrollkeeper DB in '/usr/var' and '/var' :P

This will probably be autodetected, thanks to Unix rights system. A package 
is always built as a normal user the first time. The usual process is then to 
add the right arguments for buildrooting in the spec file.

> > If we adopt such a solution, there has to be a caching mechanism. You
> > can't
> > rebuild it all at the startup every time.

Yup.


-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ali...@au...>]

* Eric Bischoff (eb...@cy...) wrote at 22:36 on 27/10/00:
> > I would like to point out that most packages do not add
> > '--localstatedir=/var' in their spec files :P
> >
> > So you may end up with a scrollkeeper DB in '/usr/var' and '/var' :P
> 
> This will probably be autodetected, thanks to Unix rights system. A package 
> is always built as a normal user the first time. The usual process is then to 
> add the right arguments for buildrooting in the spec file.

No it won't - most SPEC files do not do this!!
 
> > > If we adopt such a solution, there has to be a caching mechanism. You
> > > can't
> > > rebuild it all at the startup every time.
> 
> Yup.
> 
> 
> -- 
> Éric Bischoff  -  Documentation and Localization
> Caldera (Deutschland) GmbH - Linux for eBusiness
> Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
> http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Le Samedi 28 Octobre 2000 00:38, Ali Abdin a écrit :
> > This will probably be autodetected, thanks to Unix rights system. A
> > package is always built as a normal user the first time. The usual
> > process is then to add the right arguments for buildrooting in the spec
> > file.
>
> No it won't - most SPEC files do not do this!!

I suppose this is a misunderstanding. Yes they do. They need it at least to 
pass the build root to the make files.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Mon, 23 Oct 2000, laszlo kovacs wrote:

> > Would it be possible to change the database using an environment
> > variable? ;)
> > 
> > Secondly, how about .spec files set an environment variable to
> > OMF_NO_INSTALL when building? (and unset it when done) - Then
> > scrollkeeper can detect it and just not install the file into its
> > database.
> > 
> > This would work for tarball users too :)
> > 
> > Sure, there are corner cases where it won't work (compiling an RPM in the
> > background and compiling a tarball in the foreground (with rougly
> > similiar install times)). But I don't like corner cases :P
> I dont know if this breaks Eric's principle of doing the same thing by
> make install when is run for package creation or when it is run for
> install. For Solaris we would still need post install scripts. I would
> go for either post install scripts or partial databases, but we should
> keep it simple, so I dont like the idea of using different approaches
> for different package formats. 

What exactly is the proposal? I don't understand how post-install or 
partial databases could fix the problem?

Perhaps we can really do a normal install of the OMF file. When 
ScrollKeeper access the metadata it would check that the file "really 
exists" if it does then okay, if it doesn't it gets deleted from the 
database and "not found" is returned.

> > I don't think you should pass the XML DOM tree. I think there should be
> > API functions (or commands or whatever) to allow you to get every item in
> > the tree. Also, to speed things up, there should be a cache (so you don't
> > have to re-build the XML DOM tree all the time).
> Maybe. I was trying to stress the importance of presenting a unified
> database rather than lots of partial databases to the help browser (in
> case Scrollkeeper output would be partial databases). The implementation
> is not that important at the moment.

I agree.

Regards,
Ali

Re: [Scrollkeeper-devel] What about the distributions?

[laszlo k. <las...@su...>]

> What exactly is the proposal? I don't understand how post-install or
> partial databases could fix the problem?
I think the problem was described at the beginning of the proposal. In
order to install a doc with Scrollkeeper, a script has to be run that
changes the Scrollkeeper database. If this is done from "make install"
then the database will be modified when a package is installed with
"make install" or when the package is turned to another format (like
RPM) as the conversion process also uses "make install". We dont want
the later. Eric thinks "make install" should behave indentically when
run from command line (for installation) or when used to create a
package in a different format. He also thinks that running Scrollkeeper
install from a post install script is not an option as then post install
scripts have to be added to a large number of packages and that would
upset packagers and post install scripts are slow. He suggests that we
should use partial databases when we build a package (or install with
make install, I am not sure, but I dont think it is important). The
partial database would contain the data of that package's documentation.
This would be exported to the help browser. So the help browser has to
deal with partial databases on its own or through a library we supply to
create an interface that would unify the partial databases for the
browser.

> Perhaps we can really do a normal install of the OMF file. When
> ScrollKeeper access the metadata it would check that the file "really
> exists" if it does then okay, if it doesn't it gets deleted from the
> database and "not found" is returned.
I thought the file would be installed in /tmp at RPM creation so
Scrollkeeper would think it is installed. Or this is not what you are
talking about?

Laszlo

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

laszlo kovacs wrote:
> 
> > What exactly is the proposal? I don't understand how post-install or
> > partial databases could fix the problem?
> I think the problem was described at the beginning of the proposal. In
> order to install a doc with Scrollkeeper, a script has to be run that
> changes the Scrollkeeper database. If this is done from "make install"
> then the database will be modified when a package is installed with
> "make install" or when the package is turned to another format (like
> RPM) as the conversion process also uses "make install". We dont want
> the later. Eric thinks "make install" should behave indentically when
> run from command line (for installation) or when used to create a
> package in a different format. He also thinks that running Scrollkeeper
> install from a post install script is not an option as then post install
> scripts have to be added to a large number of packages and that would
> upset packagers and post install scripts are slow. He suggests that we
> should use partial databases when we build a package (or install with
> make install, I am not sure, but I dont think it is important). The
> partial database would contain the data of that package's documentation.
> This would be exported to the help browser. So the help browser has to
> deal with partial databases on its own or through a library we supply to
> create an interface that would unify the partial databases for the
> browser.

Great summary! ;-)

---------------- user directly installing ---------
USER	        make install => one of many small databases or one big
database,
			        as we wish, depending on whether the name depends on the
package or not
---------------- distributions --------------------
PACKAGER	make-install => partial database
USER		partial database => copied (=> small databases) or merged (=> big
database)

> I thought the file would be installed in /tmp at RPM creation so
> Scrollkeeper would think it is installed. Or this is not what you are
> talking about?

Yes, make install normally should not realize he's working in /tmp

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ali...@au...>]

* laszlo kovacs (las...@su...) wrote at 15:58 on 23/10/00:
> > What exactly is the proposal? I don't understand how post-install or
> > partial databases could fix the problem?
> I think the problem was described at the beginning of the proposal. In
> order to install a doc with Scrollkeeper, a script has to be run that
> changes the Scrollkeeper database. If this is done from "make install"
> then the database will be modified when a package is installed with
> "make install" or when the package is turned to another format (like
> RPM) as the conversion process also uses "make install". We dont want
> the later. Eric thinks "make install" should behave indentically when
> run from command line (for installation) or when used to create a
> package in a different format. He also thinks that running Scrollkeeper
> install from a post install script is not an option as then post install
> scripts have to be added to a large number of packages and that would
> upset packagers and post install scripts are slow. He suggests that we
> should use partial databases when we build a package (or install with
> make install, I am not sure, but I dont think it is important). The
> partial database would contain the data of that package's documentation.
> This would be exported to the help browser. So the help browser has to
> deal with partial databases on its own or through a library we supply to
> create an interface that would unify the partial databases for the
> browser.
> 
> > Perhaps we can really do a normal install of the OMF file. When
> > ScrollKeeper access the metadata it would check that the file "really
> > exists" if it does then okay, if it doesn't it gets deleted from the
> > database and "not found" is returned.
> I thought the file would be installed in /tmp at RPM creation so
> Scrollkeeper would think it is installed. Or this is not what you are
> talking about?

Yes, but when the RPM is "done building" it cleans up after itself so the file
would no longer exist under that location.

Perhaps what we need are two directories or something. So you would have:
prefix/var/lib/scrollkeeper/processed
prefix/var/lib/scrollkeeper/unprocessed

An "initial" scrollkeeper-install would put the file in the unprocessed
section, scrollkeeper could then at some designated time, check all files in
the unprocessed section, if they are found on the local drive they are moved
to processed, otherwise they are removed from unprocessed.

But this presents another issue. What about "online docs"? the above
implementation would force people to have docs on the "local" path!!! Instead
of deleting the file, perhaps we should move it into a "not-installed"
directory. That way if the help file is located at
"http://www.doc.com/file.sgml" We could access it from the not-installed
section.

The problem would arise with "syncing" the two directories (what if you have a
"not-installed" doc and then later install it locally)


So perhaps we shouldn't have multiple databases (or directories, or whatever
you want to call it). Perhaps we should always install files...

If a doc is not found/installed locally then ScrollKeeper should handle it.
This sounds a lot like what Eric was saying doesn't it? ;) But he makes it
sound that applications (help browsers) should handle it. Instead, I think
that ScrollKeeper should natively handle it. 

If I ask the ScrollKeeper database for "all installed docs" it would return
that - I /could/ ask for all "not-installed" docs (which would return these
docs + the HTTP ones + FTP ones + etc...)

Generally, when a help browser displays its "Index" (for Nautilus that is the
Help Sidebar) it would only display the "installed" docs...

What do you think of this proposal?

Regards,
Ali

Re: [Scrollkeeper-devel] What about the distributions?

[laszlo k. <las...@su...>]

> Yes, but when the RPM is "done building" it cleans up after itself so the file
> would no longer exist under that location.
> 
> Perhaps what we need are two directories or something. So you would have:
> prefix/var/lib/scrollkeeper/processed
> prefix/var/lib/scrollkeeper/unprocessed
> 
> An "initial" scrollkeeper-install would put the file in the unprocessed
> section, scrollkeeper could then at some designated time, check all files in
> the unprocessed section, if they are found on the local drive they are moved
> to processed, otherwise they are removed from unprocessed.
> 
> But this presents another issue. What about "online docs"? the above
> implementation would force people to have docs on the "local" path!!! Instead
> of deleting the file, perhaps we should move it into a "not-installed"
> directory. That way if the help file is located at
> "http://www.doc.com/file.sgml" We could access it from the not-installed
> section.
> 
> The problem would arise with "syncing" the two directories (what if you have a
> "not-installed" doc and then later install it locally)
> 
> So perhaps we shouldn't have multiple databases (or directories, or whatever
> you want to call it). Perhaps we should always install files...
> 
> If a doc is not found/installed locally then ScrollKeeper should handle it.
> This sounds a lot like what Eric was saying doesn't it? ;) But he makes it
> sound that applications (help browsers) should handle it. Instead, I think
> that ScrollKeeper should natively handle it.
> 
> If I ask the ScrollKeeper database for "all installed docs" it would return
> that - I /could/ ask for all "not-installed" docs (which would return these
> docs + the HTTP ones + FTP ones + etc...)
> 
> Generally, when a help browser displays its "Index" (for Nautilus that is the
> Help Sidebar) it would only display the "installed" docs...
> 
> What do you think of this proposal?
> 
> Regards,
> Ali

I dont think I managed to understand this proposal. Does it adress the
problem of post install scripts? We can install docs to a unified
Scrollkeeper database only through a post install script or let the help
browser to deal with the individual metadata of each package separately.
Is this adressed in the proposal above?

Laszlo

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> What exactly is the proposal? I don't understand how post-install or
> partial databases could fix the problem?

When the packager does "make install" on his build machine, then
scrollkeeper finds no existing database because usually it is building
in
/tmp.

Scrollkeeper has three options in front of a non-existing database:
- exit with an error code. Bade, it means that the build has failed
- do nothing, but then we lose the database information it wanted to
store
- create a new database and store into it the partial information
  (i.e. the information relative to this package's doc files)

Suppose we take the third solution. Now we have a partial database. What
do we
do with it?

- the SPEC file can simply copy it, with a unique name, on the user's
hard disk. But then the browser has to cope with that, perharps with
some help from scrollkeeper

- or scrollkeeper can merge it with the existing big single-file
database on the user's hard disk. Technically simple and efficient, but
has two drawbacks: it obligates the distribution packagers to modify the
SPEC files, and it slows down the post-installation phase.

> Perhaps we can really do a normal install of the OMF file. When
> ScrollKeeper access the metadata it would check that the file "really
> exists" if it does then okay, if it doesn't it gets deleted from the
> database and "not found" is returned.

The RPM system assumes that what you do during a "make install" is
exactly the same whether the end user does it or the packager does it.
The only difference is buildrooting (i.e. / => /tmp)

> > Maybe. I was trying to stress the importance of presenting a unified
> > database rather than lots of partial databases to the help browser (in
> > case Scrollkeeper output would be partial databases). The implementation
> > is not that important at the moment.
> 
> I agree.

The implementation is not important, but the problem that we *might
have* partial databases must be addressed.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ali...@au...>]

* Eric Bischoff (eb...@cy...) wrote at 23:12 on 25/10/00:
> Ali Abdin wrote:
> > 
> > What exactly is the proposal? I don't understand how post-install or
> > partial databases could fix the problem?
> 
> When the packager does "make install" on his build machine, then
> scrollkeeper finds no existing database because usually it is building
> in
> /tmp.

You can figure out where the Scrollkeeper databasebase is easily (just add a
'scrolkeeper-config' script so 'scrollkeeper-config --prefix'/lib/scrollkeeper
would be the database location)
 
> Scrollkeeper has three options in front of a non-existing database:
> - exit with an error code. Bade, it means that the build has failed
> - do nothing, but then we lose the database information it wanted to
> store
> - create a new database and store into it the partial information
>   (i.e. the information relative to this package's doc files)
> 
> Suppose we take the third solution. Now we have a partial database. What
> do we
> do with it?
> 
> - the SPEC file can simply copy it, with a unique name, on the user's
> hard disk. But then the browser has to cope with that, perharps with
> some help from scrollkeeper
> 
> - or scrollkeeper can merge it with the existing big single-file
> database on the user's hard disk. Technically simple and efficient, but
> has two drawbacks: it obligates the distribution packagers to modify the
> SPEC files, and it slows down the post-installation phase.
> 
> > Perhaps we can really do a normal install of the OMF file. When
> > ScrollKeeper access the metadata it would check that the file "really
> > exists" if it does then okay, if it doesn't it gets deleted from the
> > database and "not found" is returned.
> 
> The RPM system assumes that what you do during a "make install" is
> exactly the same whether the end user does it or the packager does it.
> The only difference is buildrooting (i.e. / => /tmp)
> 
> > > Maybe. I was trying to stress the importance of presenting a unified
> > > database rather than lots of partial databases to the help browser (in
> > > case Scrollkeeper output would be partial databases). The implementation
> > > is not that important at the moment.
> > 
> > I agree.
> 
> The implementation is not important, but the problem that we *might
> have* partial databases must be addressed.

Partial databases would suck. basically you are saying a "database per file"
which would suck if you have hundreds of files! (imagine Scrollkeeper has to
open each file, read from it, build XML tree, close file, extract metadata).
It would be much more efficient to do this over one file.

I think forcing people to use a post-install spec-file is not as "bad" as the
alternative.

You "could" turn scrollkeeper into a daemon that automatically merges "partial
databases" into a big one, or you "could" install a cron script. But to me
these are even worse solutions than just making people add an extra script to
the spec file.

Your arguments for no script in the spec file are:

1) It'll take more time for the RPM or DEB to install due to the "slow"
post-install/post-uninstall scripts

- oh wow, an extra couple of seconds for the RPM or deb to install/uninstall???
  I don't think this is that big of a deal

2) People are not going to want to change the spec files. 

- Well, if people want to use scrollkeeper, then they will have to write an
  OMF file, hack the Makefile.am's and possibly configure.in's - What would be
  wrong with adding the spec file to this list too? If you want your docs to
  utilize scrollkeeper, then you will have to support it.

Any other points?

Regards,
Ali

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> 
> Perhaps what we need are two directories or something. So you would have:
> prefix/var/lib/scrollkeeper/processed
> prefix/var/lib/scrollkeeper/unprocessed

This is great!!!! :-)

a) during "make install"
"scrollkeeper --install" will stores the partial information in either
/var/lib/scrollkeeper/unprocessed (end-user doing "make install")
or in
/tmp/var/lib/scrollkeeper/unprocessed (packager building the package)

b) during RPM installation
then the unprocessed file is simply copied into
/var/lib/scrollkeeper/unprocessed

c) when the help browser starts
it is requested to launch "scrollkeeper --update". This one sees that
they are unprocessed files,
merges them into the real database, and removes them from the
/unprocessed directory.

In fact the "unprocessed" directory should be two directories:

	/var/lib/scrollkeeper/toinstall
	/var/lib/scrollkeeper/touninstall

because we're having the same problem during uninstallation.

> An "initial" scrollkeeper-install would put the file in the unprocessed
> section, scrollkeeper could then at some designated time, check all files in
> the unprocessed section, if they are found on the local drive they are moved
> to processed, otherwise they are removed from unprocessed.
> 
> But this presents another issue. What about "online docs"? the above
> implementation would force people to have docs on the "local" path!!! Instead
> of deleting the file, perhaps we should move it into a "not-installed"
> directory. That way if the help file is located at
> "http://www.doc.com/file.sgml" We could access it from the not-installed
> section.
> 
> The problem would arise with "syncing" the two directories (what if you have a
> "not-installed" doc and then later install it locally)
> 
> So perhaps we shouldn't have multiple databases (or directories, or whatever
> you want to call it). Perhaps we should always install files...
> 
> If a doc is not found/installed locally then ScrollKeeper should handle it.
> This sounds a lot like what Eric was saying doesn't it? ;) But he makes it
> sound that applications (help browsers) should handle it. Instead, I think
> that ScrollKeeper should natively handle it.
> 
> If I ask the ScrollKeeper database for "all installed docs" it would return
> that - I /could/ ask for all "not-installed" docs (which would return these
> docs + the HTTP ones + FTP ones + etc...)
> 
> Generally, when a help browser displays its "Index" (for Nautilus that is the
> Help Sidebar) it would only display the "installed" docs...
> 
> What do you think of this proposal?

Great. I'm not sure about the online docs, but for the local docs it's
really a good idea.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Le Jeudi 26 Octobre 2000 01:21, vous avez écrit :
>
> Partial databases would suck. basically you are saying a "database per
> file" which would suck if you have hundreds of files! (imagine Scrollkeeper
> has to open each file, read from it, build XML tree, close file, extract
> metadata). It would be much more efficient to do this over one file.

We can build a single one out of the "not-installed-yet" partial databases.

> I think forcing people to use a post-install spec-file is not as "bad" as
> the alternative.

Those people are distribution packagers. So far OMF and scrollkeeper is a big 
nothing in the linux world, and you would want them to change all their spec 
files? Even if they agreed they would not have the time.

> You "could" turn scrollkeeper into a daemon that automatically merges
> "partial databases" into a big one, or you "could" install a cron script.
> But to me these are even worse solutions than just making people add an
> extra script to the spec file.

You can't "make" people do what you want. If they don't want it, they won't 
do it.

> Your arguments for no script in the spec file are:
>
> 1) It'll take more time for the RPM or DEB to install due to the "slow"
> post-install/post-uninstall scripts
>
> - oh wow, an extra couple of seconds for the RPM or deb to
> install/uninstall??? I don't think this is that big of a deal

Right. This is not the real point.

> 2) People are not going to want to change the spec files.
>
> - Well, if people want to use scrollkeeper, then they will have to write an
>   OMF file, hack the Makefile.am's and possibly configure.in's - What would
> be wrong with adding the spec file to this list too?

It's not the same people!!!!

The developper will write the OMF file, hack Makefile.am and configure.in. 
It's a nothing in front of what a development task is.

The distribution packager will just ignore those new files. He will probably 
don't even notice they are there. They will probably simply get copied with 
the HTML files without he notices them. He will probably not even know 
scrollkeper exists.

> If you want your docs
> to utilize scrollkeeper, then you will have to support it.

"you" in the above sentence is two different persons.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ali...@au...>]

* Eric Bischoff (eb...@cy...) wrote at 22:28 on 27/10/00:
> Le Jeudi 26 Octobre 2000 01:21, vous avez écrit :
> >
> > Partial databases would suck. basically you are saying a "database per
> > file" which would suck if you have hundreds of files! (imagine Scrollkeeper
> > has to open each file, read from it, build XML tree, close file, extract
> > metadata). It would be much more efficient to do this over one file.
> 
> We can build a single one out of the "not-installed-yet" partial databases.

When do we do the "building" though? It is NOT acceptable to do the merging
whenever we do an API function or whenever the "browser starts".

We "could" make scrollkeeper a daemon (like oafd and gconfd). We could then
have a 'scrollkeeper_init()' API function that would check if the daemon is
running (if its not, it would execute it). You could also have a
is_scrollkeeper_daemon_running or something. I think this is very similiar to
the way oafd is done.

The daemon would "merge" the not-installed-yet to the "single one" when
started and every <X> amount of time 

This is up to Laszlo though.
 
> > I think forcing people to use a post-install spec-file is not as "bad" as
> > the alternative.
> 
> Those people are distribution packagers. So far OMF and scrollkeeper is a big 
> nothing in the linux world, and you would want them to change all their spec 
> files? Even if they agreed they would not have the time.
>
> > You "could" turn scrollkeeper into a daemon that automatically merges
> > "partial databases" into a big one, or you "could" install a cron script.
> > But to me these are even worse solutions than just making people add an
> > extra script to the spec file.
> 
> You can't "make" people do what you want. If they don't want it, they won't 
> do it.

Exactly - people have to /want/ to use Scrollkeeper

> > Your arguments for no script in the spec file are:
> >
> > 1) It'll take more time for the RPM or DEB to install due to the "slow"
> > post-install/post-uninstall scripts
> >
> > - oh wow, an extra couple of seconds for the RPM or deb to
> > install/uninstall??? I don't think this is that big of a deal
> 
> Right. This is not the real point.
> 
> > 2) People are not going to want to change the spec files.
> >
> > - Well, if people want to use scrollkeeper, then they will have to write an
> >   OMF file, hack the Makefile.am's and possibly configure.in's - What would
> > be wrong with adding the spec file to this list too?
> 
> It's not the same people!!!!
> 
> The developper will write the OMF file, hack Makefile.am and configure.in. 
> It's a nothing in front of what a development task is.
> 
> The distribution packager will just ignore those new files. He will probably 
> don't even notice they are there. They will probably simply get copied with 
> the HTML files without he notices them. He will probably not even know 
> scrollkeper exists.
> 
> > If you want your docs
> > to utilize scrollkeeper, then you will have to support it.
> 
> "you" in the above sentence is two different persons.

If you speaking about Debian, the package author is "usually" different than
the packager.

Unfortunately (or fortunately?) it is usually the opposite with RPMS. The
author also handles the .spec files

Regards,
Ali

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Le Samedi 28 Octobre 2000 00:34, Ali Abdin a écrit :
> > We can build a single one out of the "not-installed-yet" partial
> > databases.
>
> When do we do the "building" though? It is NOT acceptable to do the merging
> whenever we do an API function or whenever the "browser starts".
>
> We "could" make scrollkeeper a daemon (like oafd and gconfd). We could then
> have a 'scrollkeeper_init()' API function that would check if the daemon is
> running (if its not, it would execute it). You could also have a
> is_scrollkeeper_daemon_running or something. I think this is very similiar
> to the way oafd is done.
>
> The daemon would "merge" the not-installed-yet to the "single one" when
> started and every <X> amount of time
>
> This is up to Laszlo though.

If one examines how the existing help environments do, he notices that:
- man pages have a cron daemon which I suppose does some reindexing
- info pages require you to call an install-info utility from the postinstall 
and postuninstall scripts (and this sucks)

No one calls the reindexing function from the browser. This tends to confirm 
what you say.

> > You can't "make" people do what you want. If they don't want it, they
> > won't do it.
>
> Exactly - people have to /want/ to use Scrollkeeper

What do you do for that? Light candles and pray?

Let's be serious. No one will want to adapt thousand of spec files for a new 
help indexing system.

> > > If you want your docs
> > > to utilize scrollkeeper, then you will have to support it.
> >
> > "you" in the above sentence is two different persons.
>
> If you speaking about Debian, the package author is "usually" different
> than the packager.
>
> Unfortunately (or fortunately?) it is usually the opposite with RPMS. The
> author also handles the .spec files

????? This is simply completly wrong.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Rich M. <rd...@cf...>]

>>  > You can't "make" people do what you want. If they don't want it, they
>>  > won't do it.
>>
>>  Exactly - people have to /want/ to use Scrollkeeper
>
>What do you do for that? Light candles and pray?
>
>Let's be serious. No one will want to adapt thousand of spec files for a new
>help indexing system.

Several counterexamples come to mind.  In the Bad Old Days, most C files
contained ifdefs of the form "#ifdef SunOS", containing semi-repetitive
blocks of code.  This has mostly gone away, thanks to Configure, et al.
Nobody forced folks to adopt the new technology; it simply won because it
served a perceived need and wasn't _that_ hard to use.  Eventually, the
force of public opinion became strong enough to convince many holdouts to
convert.  If you build it RIGHT, they will come.

Also, though this is not a perfect solution, you should allow folks who
are not the original developers to create file characterizations, etc.
This way, any hacker can spend an evening earning a few egoboo points
(and perhaps learning a bit) by annotating an existing package.  I have
some flaky-sounding ideas about formalizing egoboo points, if anyone is
interested...

-r
-- 
--
Rich Morin:          rd...@cf..., +1 650-873-7841, http://www.cfcl.com/rdm
Prime Time Freeware: in...@pt..., +1 408-433-9662, http://www.ptf.com
MacPerl: http://www.macperl.com, http://www.ptf.com/ptf/products/MPPE
MkLinux: http://www.mklinux.org, http://www.ptf.com/ptf/products/MKLP