refdb-users — List for RefDB users

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Torsten B. <br...@ph...>]

Hallöchen!

Markus Hoenicka writes:

> Torsten Bronger writes:
>
>> It bases on SVN 471.
>
> This is weird as there were no changes between revisions 470 (the
> last change to refdbd.c before 471) and HEAD in the relevant
> libdbi code. refdbd built from revision 471 should not have any
> problems starting without a main database. I suspect refdbd fails
> to start for other reasons.

I thought it fails on Stefan's system because it doesn't find
libdbd-sqlite3.  This was the reason why I suggested to swap
libdbd-sqlite3 and libdbd-sqlite in the "Depends:" and "Recommends":
fields of the package.  Which libdbd's must be installed for a
refdbd which doesn't find any configuration files?

For me, it works, probably because my system has everything
installed anyway.

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Torsten Bronger writes:
 > It bases on SVN 471.
 > 

This is weird as there were no changes between revisions 470 (the last
change to refdbd.c before 471) and HEAD in the relevant libdbi
code. refdbd built from revision 471 should not have any problems
starting without a main database. I suspect refdbd fails to start for
other reasons. Does refdbd create any log output when the package
attempts to start the daemon? If the default log file is not writable,
it should fall back to syslog and leave traces there.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Torsten B. <br...@ph...>]

Hallöchen!

Markus Hoenicka writes:

> [...]
>
> I'm afraid I've asked before, but are these packages based on a
> tarball or on the current svn version?

It bases on SVN 471.

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Stefan S. <ste...@ya...>]

> Markus Hoenicka writes:
> Just to make sure: does refdbd run, i.e. is there a refdbd
> process
> running after the Debian and Ubuntu package installations
> even if it
> returns only the error message mentioned above?

No refdbd process is running after the Debian package is installed.

Kind regards

Stefan


      

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Quoting Stefan Schlee <ste...@ya...>:

>> Markus Hoenicka writes:
>> Just to make sure: does refdbd run, i.e. is there a refdbd
>> process
>> running after the Debian and Ubuntu package installations
>> even if it
>> returns only the error message mentioned above?
>
> No refdbd process is running after the Debian package is installed.
>

I'm afraid I've asked before, but are these packages based on a  
tarball or on the current svn version?

regards,
Markus


-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Torsten Bronger writes:
 > My idea was to have refdbd at least running, even if in a precarious
 > mode of operation.  The problem is that Debian's packaging tool
 > wants to start refdbd after the installation and if this fails, the
 > whole installation is marked to have failed.  It's okay if refdbd
 > doesn't do anything but emitting error messages if connected, but it
 > must run.
 > 

I went back reading the code. Looks like refdbd already works the way
it should. Earlier versions of refdbd checked at startup whether or
not the main database was present and intact. I've moved this check to
the child, so all the parent currently checks is the presence of the
requested driver. If the requested driver can be loaded successfully,
it is possible to start the daemon without further setup. It will send
an error message to each client complaining about the missing main
database until you create it.

Just to make sure: does refdbd run, i.e. is there a refdbd process
running after the Debian and Ubuntu package installations even if it
returns only the error message mentioned above?

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Quoting Stefan Schlee <ste...@ya...>:

>> Ubuntu version as "derived" because this is the normal case.  We put
>> both in a repo that both of us have access to, and if others take
>> over maintainance, the nuisance is minimal.

Feel free to use the RefDB svn repository to maintain your package  
data if that suits your needs. I'd just have to add you as developers.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Stefan S. <ste...@ya...>]

Hallo!

Torsten Bronger writes:

> But I still don't think that in any phase it is good to start refdbd
> as root.  After all, even when initialising, refdbd writes Sqlite
> files that it must write later as well.

I agree, the less is done as root the better.


> When you have time, we'll try to get both packages as equal as
> possible, then we declare the Debian version as "original" and the
> Ubuntu version as "derived" because this is the normal case.  We put
> both in a repo that both of us have access to, and if others take
> over maintainance, the nuisance is minimal.

Fine, I am looking forward to!

Kind regards

Stefan



      

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Torsten Bronger writes:
 > By the way, there are also other things to sort out.  For example,
 > generating the excellent RefDB documentation was a nightmare.  To my
 > surprise, dtdparse is not available on Ubuntu.  Then, FOP simply
 > crashs.  I suspect a bug in the current FOP version, so we have to
 > wait for the next one and hope for the best.  And finally, I think

I just tried building the manual with fop-0.95 which results in the
crash that you mentioned. I went ahead and built the current svn
revision of fop which works just fine. This problem should vanish once
the next fop release is available.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Quoting Stefan Schlee <ste...@ya...>:

>>
>> I don't like that.  Unless there are very good reasons, a daemon
>> should have non-root privileges.  RefDB works fine as an ordinary
>> user.
>
> The manual says: "The interactive script refdb-init must be run with  
> root permission as it tries to fiddle with a couple of files that a  
> regular user should not have write permission for."  So starting  
> "refdb-init" as root seems to be necessary.  This does not mean that  
> the "refdb" server runs as root.  As I understand it, the command  
> "/usr/sbin/refdbd" (called as $MYREFDBD) is merely used to set up  
> the "refdb" database.  The refdbd server is started after that by  
> the command
>
> $MYREFDBCTL start || endScript "Could not start refdbd" "failed"
>
> which starts it with the "refdb" user (see the variable DAEMONUSER  
> in the "/etc/init.d/refdb" script).
>

Basically, refdbd runs just fine with regular user permissions as long  
as you set up all file permissions appropriately. The manual has  
generic installation instructions which make sure that you arrive at a  
working installation. If you have extra security concerns (like Debian  
has, for good reasons), then you'll have to adapt these instructions  
accordingly. If you run refdbd using a different user account, it  
should be sufficient to make all files that refdbd accesses (mainly  
log and PID files) writable for that user. It is very well possible  
that refdb-init requires additional tweaks to make it suitable for  
non-root accounts. I've simply never tried that. I'll try this as soon  
as time permits.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Torsten B. <br...@ph...>]

Hallöchen!

Stefan Schlee writes:

> Torsten Bronger writes:
>
> [...]
>
>>>  -- I had to execute: "/usr/sbin/refdbd" as root and change the
>>> ownership of the "/var/lib/refdb/db" hierarchy afterwards.
>>
>> Then I have to fix this because this must not be necessary.
>> Probably it's a Sqlite config issue because with Postgres, the
>> limited privileges work fine, including properly writing the PID
>> and the log file.
>
> Simply remove the 09_refdbd_user_always_refdb_in_refdb-init.patch
> from the Ubuntu source package, the original script is ok:
>
> # create the main database
> if [ "${engine_name}" = "pgsql" ]; then
>     su $postgres_name -c "$MYREFDBD -a -e 0 -l 6 -u $dbadmin_name -w
> $dbadmin_password" || endScript "Check for PostgreSQL authentication problems"
> "failed"
> else
>     $MYREFDBD -a -e 0 -l 6 -u $dbadmin_name -w $dbadmin_password || endScript
> "Failed to install or upgrade main database" "failed"
> fi
>
> which means that in the sqlite(3) case the "/usr/sbin/refdb" is
> run as root.

Okay, I see that root is necessary to write the RefDB config files.

But I still don't think that in any phase it is good to start refdbd
as root.  After all, even when initialising, refdbd writes Sqlite
files that it must write later as well.

>>> ...
>> 
>> No, PID and log files are set to permissions which allow the
>> "refdb" user to write them.
>
> You are right, the permissions are set correctly.  But removing
> the above patch does not completely solve the problem for me, I
> have to execute
>
> chown -R refdb:refdb /var/lib/refdb/db
>
> after the "refdb" database is set up.

That's the problem.  I must learn Sqlite better; I'll do it this
weekend.  Initialising a database should not require more file
system privileges than writing to it.  If Sqlite doesn't agree with
me, I am disappointed.

> [...]
>
>>>  - I do not know which of the original shortcomings have already
>>> been addressed in a possible new version of Torsten's Ubuntu
>>> source package.  As I am building and installing in a Debian
>>> context, I do not know if these (or which of these) shortcomings
>>> should be handled in an Ubuntu package.
>> 
>> I didn't do anything except fixing the wrong MYREFDBCTL path above.
>> I won't have time for it until the weekend.
>> 
>> Anyway, the Ubuntu package must be built in a way that only minimal
>> modifications are necessary for Debian.  
>
> Full ack.
>
>> I don't have any experience with packages beyond packaging RefDB,
>> let alone with packaging for Debian.  How is it done normally?  Is
>> there one "source" package (my Ubuntu package in this case) and one
>> "derived" package with a couple of meta-patches (your Debian package
>> in this case)?  How do packagers organise this?
>
> [...]
>
> The definitive sources concerning packaging for Debian are:
>
>  - the "Debian New Maintainers' Guide"
> (http://www.debian.org/doc/manuals/maint-guide/)
>
>  - the "Debian Developer's Reference"
> (http://www.debian.org/doc/manuals/developers-reference/) and
>
>  - the  "Debian Policy Manual" (http://www.debian.org/doc/debian-policy/).  

The Ubuntu people must read the same docs since the differences are
small to non-existent.  Normally, it's the other way round, i.e. the
Debian package is first.

When you have time, we'll try to get both packages as equal as
possible, then we declare the Debian version as "original" and the
Ubuntu version as "derived" because this is the normal case.  We put
both in a repo that both of us have access to, and if others take
over maintainance, the nuisance is minimal.

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Stefan S. <ste...@ya...>]

Hallo!

Markus Hoenicka writes:

> As mentioned in my reply to Torsten, the server could be modified to
> start but return error messages to each client request as long as the
> main database has not been created.

Good solution!

> >  - After the binary installation I started the "refdb-init" script as root.  To execute this script successfully I had to make two minor modifications (Torsten already notes difficulties concerning this script in the README.Debian file of the package.)
> >
> >  -- I had the change the value of MYREFDBCTL to "/etc/init.d/refdb"
> >
> >  -- I had to execute: "/usr/sbin/refdbd" as root and change the ownership of the "/var/lib/refdb/db" hierarchy afterwards.
> >
> 
> I guess this amount of porting is acceptable for a binary package. I
> assume that Debian packages use the same sort of patch collections as
> BSD ports do to cope with the idiosyncrasies of these OSes.

As Torsten writes he will handle the first point in the Ubuntu source package. The first half of the second point should - from my point of view - also be handled by Torsten, by removing a patch from the Ubuntu source package.  Still I wonder how the second half of the second point is handled for other installations.  If "/usr/sbin/refdbd" sets-up the "refdb" database as "root", the refdb-file will be owned by root (sqlite(3) case).  Then of course the "refdbd"-server can only write to the "refdb"-database if it executes as root (because the permissions are set to 644).  As Torsten notes a server that executes as "root" is a possible security breach.  Hence - at least in the Ubuntu/Debian installatin of the refDB software the "refdb"-server executes as user "refdb".

As I understand, in the original refdb-init script the "refdbd"-servers is startd by the "refdbctl" script, which is not used in the Ubuntu/Debian installation.  It seems to me that according to this scripts the "refdb"-server executes as "root".  Then of course there are no access problems, but possible security problems if the server gets corrupted.  Maybe this should be changed upstream; but maybe it is handled differently anyway and I am not aware of it.


>> After fiddling with the refdb-init script I have to questions concerning this script:
>>
>>  - I do not understand why in the call to refdbd contains the -e flag set to 0 AND the -L flag set.  Shouldn't the -e flag be set to 2 in this case or the -L flag be left out?  But maybe this means falling back to stderr if the file is not accessible, which is not exactly what the manual says.
>>
> 
> Which version of RefDB do you use? Actually, I can't find the -L flag
> being used with refdbd in the current svn revision of refdb-init.in

I inspected the source tree _after_ building the binary packages.  This led me to the false belief that I was reading the refdb-init in its upstream version, when I actually read it in its version patched by the Ubuntu source package.  Sorry for that.

>> - In the refdb-init.in file most of the path-names (for example the
>>      path-name to the database-directory) are made configurable in the
>>      build-process, but this is not the case for the
>>      example-configuration files.  In the way the software is
>>      currently setup there should not arise any inconsistencies.
>>      Nevertheless a cleaner solution seems to me to make the
>>      example-configuration files configurable in the build-process and
>>      to let the refdb-init script read these values from the selected
>>      configuration file.
> 
> I'm not sure whether I can follow you here. The example config files
> are built from input files. All refdbdrc example config files are
> built from the same source refdbdrc. Which variables in particular are
> not adjusted properly? I may have missed something which happens to
> work on FreeBSD but not on other OSes.

In the refdb-init script you check whether a reference database of the name the user inputs already exist.  In the case of an sqlite(3) database the pertaining line of code reads as follows:

ls <db_path>/$referencedb_name 2>/dev/null \
		&& echo "Reference database exists and I'm not going to fiddle with it. Do you want to pick a different name [y/n]?" && input=$(confirm)

What I meant is that if the administrator changed the "dbpath" entry in the "refdbdrc.sqlite(3).example" file before executing the "refdb-init" script this test would not make sense.  On the one hand this is obviously not the way you intended the administrator to proceed and it may sound like a contrived example.  On the other hand setting such data only in one place, the "etc/refdbdrc" (which gets transformed into the "refdbdrc.sqlite(3).example" file) and always reading it from this source is a principle worth adhering to, because it leads to more robust code.

Kind regards

Stefan



      

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Stefan S. <ste...@ya...>]

Hallo!

Torsten Bronger writes:

>>
>>  - After the binary installation I started the "refdb-init" script
>> as root.
>
> I don't like that.  Unless there are very good reasons, a daemon
> should have non-root privileges.  RefDB works fine as an ordinary
> user.

The manual says: "The interactive script refdb-init must be run with root permission as it tries to fiddle with a couple of files that a regular user should not have write permission for."  So starting "refdb-init" as root seems to be necessary.  This does not mean that the "refdb" server runs as root.  As I understand it, the command "/usr/sbin/refdbd" (called as $MYREFDBD) is merely used to set up the "refdb" database.  The refdbd server is started after that by the command 

$MYREFDBCTL start || endScript "Could not start refdbd" "failed"

which starts it with the "refdb" user (see the variable DAEMONUSER in the "/etc/init.d/refdb" script).


>>  -- I had to execute: "/usr/sbin/refdbd" as root and change the
>> ownership of the "/var/lib/refdb/db" hierarchy afterwards.
>
> Then I have to fix this because this must not be necessary.
> Probably it's a Sqlite config issue because with Postgres, the
> limited privileges work fine, including properly writing the PID and
> the log file.

Simply remove the 09_refdbd_user_always_refdb_in_refdb-init.patch from the Ubuntu source package, the original script is ok:

# create the main database
if [ "${engine_name}" = "pgsql" ]; then
    su $postgres_name -c "$MYREFDBD -a -e 0 -l 6 -u $dbadmin_name -w $dbadmin_password"	|| endScript "Check for PostgreSQL authentication problems" "failed"
else
    $MYREFDBD -a -e 0 -l 6 -u $dbadmin_name -w $dbadmin_password || endScript "Failed to install or upgrade main database" "failed"
fi

which means that in the sqlite(3) case the "/usr/sbin/refdb" is run as root.


>> ...
> 
> No, PID and log files are set to permissions which allow the "refdb"
> user to write them.

You are right, the permissions are set correctly.  But removing the above patch does not completely solve the problem for me, I have to execute 

chown -R refdb:refdb /var/lib/refdb/db

after the "refdb" database is set up.

Because "/usr/sbin/refdbd" runs as root the "refdb" database file that it sets-up is owned by root.  When the "refdba" command subsequently tries to add styles this is - to my understanding - done via the "refdbd"-server.  Because the "refdbd"-server runs as user "refdb" it fails to access the "refdb" database, which is owned by "root" and has permissions 644.  

In addition to that a reference database is set up with the command 

$MYREFDBA -c stdout -C createdb ${referencedb_name}

For this to succeed the "/var/lib/refdb/db" directory has to be owned by the user "refdb" for the same reason as above.


>>  - I do not know which of the original shortcomings have already
>> been addressed in a possible new version of Torsten's Ubuntu
>> source package.  As I am building and installing in a Debian
>> context, I do not know if these (or which of these) shortcomings
>> should be handled in an Ubuntu package.
> 
> I didn't do anything except fixing the wrong MYREFDBCTL path above.
> I won't have time for it until the weekend.
> 
> Anyway, the Ubuntu package must be built in a way that only minimal
> modifications are necessary for Debian.  

Full ack.

> I don't have any experience with packages beyond packaging RefDB,
> let alone with packaging for Debian.  How is it done normally?  Is
> there one "source" package (my Ubuntu package in this case) and one
> "derived" package with a couple of meta-patches (your Debian package
> in this case)?  How do packagers organise this?

Good question, but as I already mentioned  in a pre-post although I am using Debian for quite a while I am newbie concerning Debian packaging and I do not currently posses any knowledge of the interplay between Debian and Ubuntu packages.  Nevertheless, I would really love to collaborate in the building of the refdb Debian/Ubuntu package: Debian packaging is one of the top points in my current learning agenda.  But because I have very tight schedule until mid of November I wont be able to deliver any substantial input until then.

The definitive sources concerning packaging for Debian are:

 - the "Debian New Maintainers' Guide" (http://www.debian.org/doc/manuals/maint-guide/)

 - the "Debian Developer's Reference" (http://www.debian.org/doc/manuals/developers-reference/) and

 - the  "Debian Policy Manual" (http://www.debian.org/doc/debian-policy/).  

Additional important information contains the documentation of the "debconf" package/system.  

An excellent book on these topics (and more) is Martin F. Krafft's: "The Debian System"

There is an equivalent of the "Debian Policy Manual" for Ubuntu, see for example: http://people.canonical.com/~cjwatson/ubuntu-policy/policy.html/.  The Ubuntu wiki also contains valuable information, see for example "Working with Debian-format Packages" at https://wiki.ubuntu.com/UbuntuDevelopment.

Kind regards

Stefan




      

[Refdb-users] Weird iconv issue

[Markus H. <mar...@mh...>]

Torsten Bronger writes:
 > I didn't change test.ris inbetween.  Can somebody reproduce this?
 > 

Could you please post the dataset in order to reproduce this?
Actually, I don't think it is related to the title but rather to the
code which generates the citation key. Can you add any other dataset
25 times without bumping into this problem?

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[Refdb-users] Weird iconv issue

[Torsten B. <br...@ph...>]

Hallöchen!

If I have a "[D5]" in the title tag of RIS input like e.g.

TI  - Algorithm 368: Numerical inversion of Laplace transforms [D5]

addref sometimes (not always!) throws an error:

refdbc: addref test.ris
406:1:Stehfest1970s:45
408:1
999:1 added:0 skipped:0 failed
refdbc: addref test.ris
406:1:Stehfest1970t:46
408:1
999:1 added:0 skipped:0 failed
refdbc: addref test.ris
406:1:Stehfest1970u:47
408:1
999:1 added:0 skipped:0 failed
refdbc: addref test.ris
406:1:Stehfest1970v:48
408:1
999:1 added:0 skipped:0 failed
refdbc: addref test.ris
iconv: invalid input character sequence
refdbc: addref test.ris
406:1:Stehfest1970w:49
408:1
999:1 added:0 skipped:0 failed
refdbc: addref test.ris
iconv: invalid input character sequence
refdbc: addref test.ris
iconv: invalid input character sequence
refdbc: 

I didn't change test.ris inbetween.  Can somebody reproduce this?

The full RIS dataset contains only ASCII characters.

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Torsten B. <br...@ph...>]

Hallöchen!

Stefan Schlee writes:

> [...]
>
> To make a long story short: 
>
> [...]
>
>  - After the binary installation I started the "refdb-init" script
> as root.

I don't like that.  Unless there are very good reasons, a daemon
should have non-root privileges.  RefDB works fine as an ordinary
user.

> To execute this script successfully I had to make two minor
> modifications (Torsten already notes difficulties concerning this
> script in the README.Debian file of the package.)
>
>  -- I had the change the value of MYREFDBCTL to
>     "/etc/init.d/refdb"

Fixed.

>  -- I had to execute: "/usr/sbin/refdbd" as root and change the
> ownership of the "/var/lib/refdb/db" hierarchy afterwards.

Then I have to fix this because this must not be necessary.
Probably it's a Sqlite config issue because with Postgres, the
limited privileges work fine, including properly writing the PID and
the log file.

> [...]
>
>     Note: I first tried:
>
>     chown refdb:refdb /var/lib/refdb/db
>
>     su refdb -c "$MYREFDBD -a -e 0 -l 6 -L /var/log/refdb/refdbd.log -P
> /var/run/refdb/refdbd.pid -u $dbadmin_name -w $dbadmin_password" || endScript
> "Failed to install or upgrade main database" "failed"
>
>     but that did not work (denied access to the log/pid files ?)

No, PID and log files are set to permissions which allow the "refdb"
user to write them.

> [...]
>
>  - I do not know which of the original shortcomings have already
> been addressed in a possible new version of Torsten's Ubuntu
> source package.  As I am building and installing in a Debian
> context, I do not know if these (or which of these) shortcomings
> should be handled in an Ubuntu package.

I didn't do anything except fixing the wrong MYREFDBCTL path above.
I won't have time for it until the weekend.

Anyway, the Ubuntu package must be built in a way that only minimal
modifications are necessary for Debian.  I don't have any experience
with packages beyond packaging RefDB, let alone with packaging for
Debian.  How is it done normally?  Is there one "source" package (my
Ubuntu package in this case) and one "derived" package with a couple
of meta-patches (your Debian package in this case)?  How do
packagers organise this?

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

[Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Stefan Schlee writes:
 >  - I installed the binary package and accepted the error message of the post-installation script that resulted from being unable to start the refdbd server.
 > 

As mentioned in my reply to Torsten, the server could be modified to
start but return error messages to each client request as long as the
main database has not been created.

 >  - After the binary installation I started the "refdb-init" script as root.  To execute this script successfully I had to make two minor modifications (Torsten already notes difficulties concerning this script in the README.Debian file of the package.)
 > 
 >  -- I had the change the value of MYREFDBCTL to "/etc/init.d/refdb"
 > 
 >  -- I had to execute: "/usr/sbin/refdbd" as root and change the ownership of the "/var/lib/refdb/db" hierarchy afterwards. 
 > 

I guess this amount of porting is acceptable for a binary package. I
assume that Debian packages use the same sort of patch collections as
BSD ports do to cope with the idiosyncrasies of these OSes.

 > 
 > 
 > After fiddling with the refdb-init script I have to questions concerning this script:
 > 
 >  - I do not understand why in the call to refdbd contains the -e flag set to 0 AND the -L flag set.  Shouldn't the -e flag be set to 2 in this case or the -L flag be left out?  But maybe this means falling back to stderr if the file is not accessible, which is not exactly what the manual says.
 > 

Which version of RefDB do you use? Actually, I can't find the -L flag
being used with refdbd in the current svn revision of refdb-init.in

 >  - In the refdb-init.in file most of the path-names (for example the path-name to the database-directory) are made configurable in the build-process, but this is not the case for the example-configuration files.  In the way the software is currently setup there should not arise any inconsistencies.  Nevertheless a cleaner solution seems to me to make the example-configuration files configurable in the build-process and to let the refdb-init script read these values from the selected configuration file.

I'm not sure whether I can follow you here. The example config files
are built from input files. All refdbdrc example config files are
built from the same source refdbdrc. Which variables in particular are
not adjusted properly? I may have missed something which happens to
work on FreeBSD but not on other OSes.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Torsten Bronger writes:
 > 
 > There are already post-installation scripts, however, they don't
 > cover this.  So far, I only have been able to provide an
 > unconfigured RefDB.  See the README file in the debian/ directory
 > <http://bazaar.launchpad.net/~bronger/%2Bjunk/package-refdb-svn/annotate/head%3A/debian/README.Debian>.

Thanks for reminding me. I think I have perused this one before.

 > 
 > If it is possible, I suggest to rely on refdb-init instead of
 > providing a pre-installed configuration file.  A working refdb-init
 > would be very helpful.  Maybe it works already, but I don't know.
 > 
 > Theoretically, the features of refdb-init can be realised
 > interactively in the post-installation script although I dislike
 > programs prompting me after I installed them (so far, only *very*
 > few programs make use of this possibility on Ubuntu).
 > 

This is a good point as it breaks unattended installations/upgrades.

 > My idea was to have refdbd at least running, even if in a precarious
 > mode of operation.  The problem is that Debian's packaging tool
 > wants to start refdbd after the installation and if this fails, the
 > whole installation is marked to have failed.  It's okay if refdbd
 > doesn't do anything but emitting error messages if connected, but it
 > must run.
 > 

Do you know how PostgreSQL is installed on Debian? It is basically the
same two-step procedure if you do it from the sources, i.e. the
installation proper, followed by creating a database cluster, which is
a prerequisite to do anything useful with the server. If creating the
cluster is done after the package installation has finished, the RefDB
package should do just the same. In that case I'd have to make sure
the server starts even in the absence of the main database. It would
then return an error message to each client request stating this fact.

 > By the way, there are also other things to sort out.  For example,
 > generating the excellent RefDB documentation was a nightmare.  To my
 > surprise, dtdparse is not available on Ubuntu.

Surprising indeed.

 > Then, FOP simply
 > crashs.  I suspect a bug in the current FOP version, so we have to
 > wait for the next one and hope for the best.

I can't remember having any problems with FOP except that Java must be
allowed to allocate a boatload of memory. It might just be that FOP
0.95 needs even more memory than FOP 0.20. I'll look into this.

 >  And finally, I think
 > the Makefile connects to the Internet for the stylesheets (is this
 > correct?), however, this is blocked on Launchpad's servers.
 > 

This is a feature of the XSLT processor and depends on the installed
stylesheets and the contents of catalog files. You could tweak the
toolchain to make it use local copies, but then you'd have to have
them installed just in order to build the docs.

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38

[Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Stefan S. <ste...@ya...>]

Hallo!

Markus Hoenicka writes:

> Please have a look at the refdb-init script (also documented here:
> http://refdb.sourceforge.net/manual/ch04s08.html) which is a generic
> script that may or may not work out of the box on Debian.

Sigh ... big shame and a big RFM on me.  Installing software via
the Debian packaging system obviously made me lazy.

So I had a deeper look at the manual and - what a surprise - I seem to have succeeded:

After two minor modifications (see below) I seem to have a running refDB server/sqlite3-database:

The "/var/lib/refd/db" directory contains the following file:

 DB_VERSION, refdb, refs 

(where "refs" is the name of the reference DB that I have selected).  I have not yet entered a single reference into my refs reference DB, but I wanted to give you timely feedback.


To make a long story short: 

 - I installed the binary package and accepted the error message of the post-installation script that resulted from being unable to start the refdbd server.

 - After the binary installation I started the "refdb-init" script as root.  To execute this script successfully I had to make two minor modifications (Torsten already notes difficulties concerning this script in the README.Debian file of the package.)

 -- I had the change the value of MYREFDBCTL to "/etc/init.d/refdb"

 -- I had to execute: "/usr/sbin/refdbd" as root and change the ownership of the "/var/lib/refdb/db" hierarchy afterwards. 

    In concrete, instead of the original:

    su refdb -c "$MYREFDBD -a -e 0 -l 6 -L /var/log/refdb/refdbd.log -P /var/run/refdb/refdbd.pid -u $dbadmin_name -w $dbadmin_password"	|| endScript "Failed to install or upgrade main database" "failed"


    the modified version of the refdb-init script which works in my Debian context contains:

    $MYREFDBD -a -e 0 -l 6 -L /var/log/refdb/refdbd.log -P /var/run/refdb/refdbd.pid -u $dbadmin_name -w $dbadmin_password	|| endScript "Failed to install or upgrade main database" "failed"

    chown -R refdb:refdb /var/lib/refdb/db


    Note: I first tried:

    chown refdb:refdb /var/lib/refdb/db

    su refdb -c "$MYREFDBD -a -e 0 -l 6 -L /var/log/refdb/refdbd.log -P /var/run/refdb/refdbd.pid -u $dbadmin_name -w $dbadmin_password"	|| endScript "Failed to install or upgrade main database" "failed"

    but that did not work (denied access to the log/pid files ?)


After fiddling with the refdb-init script I have to questions concerning this script:

 - I do not understand why in the call to refdbd contains the -e flag set to 0 AND the -L flag set.  Shouldn't the -e flag be set to 2 in this case or the -L flag be left out?  But maybe this means falling back to stderr if the file is not accessible, which is not exactly what the manual says.

 - In the refdb-init.in file most of the path-names (for example the path-name to the database-directory) are made configurable in the build-process, but this is not the case for the example-configuration files.  In the way the software is currently setup there should not arise any inconsistencies.  Nevertheless a cleaner solution seems to me to make the example-configuration files configurable in the build-process and to let the refdb-init script read these values from the selected configuration file.

If you like I can post a quick and dirty recipe that collects all the steps for building a Debian Lenny package from the refdb_0.9.9.0~svn471-7ubuntu1 Ubuntu source package, installing and purging the Debian binary package.  This should probably only be done after I have successfully used the reference database for some time (added references, retrieved them, build bibliographies etc.)

Note though that the recipe is currently a make-shift solution:

 - The approach described in this recipe is VERY dirty indeed.  For example: I do not even remove the "ubuntu" string in the package name (this does not mean any hostility against Ubuntu :).

 - I do not know which of the original shortcomings have already been addressed in a possible new version of Torsten's Ubuntu source package.  As I am building and installing in a Debian context, I do not know if these (or which of these) shortcomings should be handled in an Ubuntu package.

Kind regards

Stefan



      

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Torsten B. <br...@ph...>]

Hallöchen!

Markus Hoenicka writes:

> Quoting Stefan Schlee <ste...@ya...>:
>
> [...]
>
>>  - In addition to that I had to incorporate a symbolic link into
>> the /etc/refdb that pointed to the appropriate configuration file
>> for this case: "ln -s refdbdrc.sqlite3.example refdbdrc".
>
> I don't know how Torsten handles these issues, but to end up with
> a working RefDB installation some post-install setup is
> mandatory. I guess that most if not all of this setup stuff could
> be handled by the package installation procedure. Please have a
> look at the refdb-init script [...]

There are already post-installation scripts, however, they don't
cover this.  So far, I only have been able to provide an
unconfigured RefDB.  See the README file in the debian/ directory
<http://bazaar.launchpad.net/~bronger/%2Bjunk/package-refdb-svn/annotate/head%3A/debian/README.Debian>.
I quote:

    After having installed RefDB, there is no configuration file.
    So RefDB works in SQLite mode.  I haven't tested yet whether
    this actually works.  For example, I don't know which file RefDB
    tries to write in this status.  Maybe it's a file in /usr/local,
    which will probably fail.

    [...]

    refdb-init is a problem.  I behaves slightly different from the
    original one because refdbd runs with the "refdb" user.
    Additionally, because we don't have refdbctrl, it calls
    /etc/init.d/refdb instead.  It must be tested more thoroughly.

If it is possible, I suggest to rely on refdb-init instead of
providing a pre-installed configuration file.  A working refdb-init
would be very helpful.  Maybe it works already, but I don't know.

Theoretically, the features of refdb-init can be realised
interactively in the post-installation script although I dislike
programs prompting me after I installed them (so far, only *very*
few programs make use of this possibility on Ubuntu).

My idea was to have refdbd at least running, even if in a precarious
mode of operation.  The problem is that Debian's packaging tool
wants to start refdbd after the installation and if this fails, the
whole installation is marked to have failed.  It's okay if refdbd
doesn't do anything but emitting error messages if connected, but it
must run.

This is the reason why I included libdbd-sqlite in the depenencies.
I thought that without configuration file, refdbd would look for
Sqlite2 bindings only.


By the way, there are also other things to sort out.  For example,
generating the excellent RefDB documentation was a nightmare.  To my
surprise, dtdparse is not available on Ubuntu.  Then, FOP simply
crashs.  I suspect a bug in the current FOP version, so we have to
wait for the next one and hope for the best.  And finally, I think
the Makefile connects to the Internet for the stylesheets (is this
correct?), however, this is blocked on Launchpad's servers.

Therefore, I excluded the documentation from the binary package for
now.

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Quoting Stefan Schlee <ste...@ya...>:

>
>  - Debian DOES distinguish between sqlite and sqlite3.  Accordingly  
> I installed the package "libdbd-sqlite3" which IS listed in  
> Torsten's source package but only under "Recommends:" and not under  
> "Depends:" (though the package "libdbd-sqlite" is listed there).
>

As mentioned previously, I'd recommend to make the package depend on  
libdbd-sqlite3. SQLite 2.x is supported only for backwards  
compatibility, but I doubt that many people use 2.x these days.

>  - In addition to that I had to incorporate a symbolic link into the  
> /etc/refdb that pointed to the appropriate configuration file for  
> this case: "ln -s refdbdrc.sqlite3.example refdbdrc".
>

I don't know how Torsten handles these issues, but to end up with a  
working RefDB installation some post-install setup is mandatory. I  
guess that most if not all of this setup stuff could be handled by the  
package installation procedure. Please have a look at the refdb-init  
script (also documented here:  
http://refdb.sourceforge.net/manual/ch04s08.html) which is a generic  
script that may or may not work out of the box on Debian. I'd greatly  
appreciate if you could provide some feedback how far this script is  
going to take you. In any case, it is supposed to create the config  
file which you found missing as well as the main database, see below.  
The config script is copied from the appropriate .example file to  
avoid overwriting local customizations during "make install".

> I created an empty test-database in the directory  
> "/var/lib/refdb/db" by issuing "sqlite3 /var/lib/refdb/db/test.db"
>
> I connected to this database with "sqlite3" and entered the  
> ".databases" command and got the result
>
> seq  name             file                                           
>            
> ---  ---------------   
> ----------------------------------------------------------
> 0    main             /var/lib/refdb/db/tes.db
>
> which seems fine.
>

Not quite. The RefDB error message "could not open main database" is  
issued if the *main* database simply isn't there, but also if it does  
not contain the expected contents. This is not a reference database,  
but an internal database where RefDB keeps bibliography styles, among  
others. Creating the main database is one of the important steps of  
the post-installation setup.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Quoting Torsten Bronger <br...@ph...>:

>
> So I should swap libdbd-sqlite3 and libdbd-sqlite between
> "Recommends" and "Depends"?
>

Yes, this is what I'd recommend. SQLite 2.x is stable but fairly old,  
and I guess most SQLite users have moved to 3.x.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Torsten B. <br...@ph...>]

Hallöchen!

Stefan Schlee writes:

> [...]
>
>> Stefan, could you please post the contents of
>> /etc/refdb/refdbdrc?  Does Debian distinguish between sqlite and
>> sqlite3?
>
> This notion has set me on the right track:
>
>  - Debian DOES distinguish between sqlite and sqlite3. 
> Accordingly I installed the package "libdbd-sqlite3" which IS
> listed in Torsten's source package but only under "Recommends:"
> and not under "Depends:" (though the package "libdbd-sqlite" is
> listed there).

So I should swap libdbd-sqlite3 and libdbd-sqlite between
"Recommends" and "Depends"?

Tschö,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: tor...@ja...
                                  or http://bronger-jmp.appspot.com

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Stefan S. <ste...@ya...>]

Hallo!

Thank you for your immediate reply!

Torsten Bronger writes:

> Du you use the latest version on
> <https://launchpad.net/~bronger/+ppa-packages>?  I switched to the
> SVN version three weeks ago.  It contains
> 
>     Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, lsb-base (>= 3.0-6),
>      openjade, docbook-dsssl, docbook-xml, jadetex, texlive-base-bin, fop,
>      libbtparse0, libtext-iconv-perl, libmarc-record-perl, libxml-parser-perl,
>      libcgi-pm-perl, coreutils, libreadline5, libxml2-utils, xml-core, libncurses5,
>      libdbi0, libdbd-sqlite, libexpat1
>     Recommends: mysql-server (>= 3.23) | postgresql (>= 7.1) | libsqlite0 (>= 2.8) |
>      libsqlite3-0 (>= 3), libdbd-mysql | libdbd-pgsql | libdbd-sqlite3

I downloaded it from <https://launchpad.net/~bronger/+archive/ppa/+packages>.  The package name with full version/revision information is : "refdb_0.9.9.0~svn471-7ubuntu1".  So, although the web-address is not exactly the same I assume that I have used the latest package build.  The contents of the "Depends:" and "Recommends:" is as you state it.

By the way, although I complaint in my original post I did not thank Torsten for providing the source package ... it has been (see below) of great help!

Markus Hoenicka writes:

> Stefan, could you please post the contents of /etc/refdb/refdbdrc?
> Does Debian distinguish between sqlite and sqlite3?

This notion has set me on the right track:

 - Debian DOES distinguish between sqlite and sqlite3.  Accordingly I installed the package "libdbd-sqlite3" which IS listed in Torsten's source package but only under "Recommends:" and not under "Depends:" (though the package "libdbd-sqlite" is listed there).

 - In addition to that I had to incorporate a symbolic link into the /etc/refdb that pointed to the appropriate configuration file for this case: "ln -s refdbdrc.sqlite3.example refdbdrc".

... and all of a sudden ... I could start the server by entering "/etc/init.d/refdb start".

Original problem solved ...

---

... next problem:

I created an empty test-database in the directory "/var/lib/refdb/db" by issuing "sqlite3 /var/lib/refdb/db/test.db"

I connected to this database with "sqlite3" and entered the ".databases" command and got the result

seq  name             file                                                      
---  ---------------  ----------------------------------------------------------
0    main             /var/lib/refdb/db/tes.db 

which seems fine.

Next  I tried to connect to the database via the refdb server by issuing "refdba".  As described in the manual I entered the "viewstat" on the command line of "refdba" to test the connection.  What I got as reply was: "could not open main databas".

In the "/etc/refdbdrc" file I used both "dbpath /var/lib/refdb/db/" and "dbpath /var/lib/refdb/db/test.db", I restarted the server for both options, always the same result "could not open main databas".

When I run refdba as "refdba -e 2 -L /home/stefan/tmp/refdba.log -l 7" it does not write additional information into the logfile.

Do you have any clue?

Thanks for your help already, kind regards 

Stefan




      

Re: [Refdb-users] Build Debian bin pkg from Ubunt scr pkg

[Markus H. <mar...@mh...>]

Quoting Torsten Bronger <br...@ph...>:

> libdbd-sqlite is always installed, however in your case, this was
> not enough.  Does anybody have an idea why is this?  Doesn't RefDB
> start in Sqlite mode?
>
> We could add *all* libdbd's to the dependencies.
>

I think it is a good idea to stick to sqlite in the first place, as  
this allows for the smallest possible installation. According to  
Stefan's refdbd log "sqlite" is available but this does not seem to be  
the driver which is requested in refdbdrc. Stefan, could you please  
post the contents of /etc/refdb/refdbdrc? Does Debian distinguish  
between sqlite and sqlite3? The latter would be a better default btw.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de