[Refdb-users] Comments on first installation of RefDB

[Markus H. <hoe...@co...>]

Hi,

first of all, thank you very much for your detailed comments on
RefDB. I appreciate your efforts to make clear what RefDB lacks from a
user's point of view.

Please find the specific comments below.

Nicolas Brouard writes:
[...]
 > working on my linux box and I want to make some comments on the
 > documentation and on the product itself which could be easily improved by a
 > better documentation (the implementation of Postgresql is a secondary
 > priority).

This is not correct. Previous user comments pointed out two major
groups of people who don't use RefDB: those who prefer PostgreSQL over
MySQL (e.g. because everything else in their offices runs on
PostgreSQL) and those who don't want to run an external database
server at all. The switch to a database abstraction layer is therefore
essential. PostgreSQL is only the first additional database server to
be supported by RefDB. A sqlite dbi driver which doesn't need any
external database server is in preparation.

A third group is those who require support for additional character
encodings, especially Unicode. The current stable MySQL versions do
not support Unicode, and the development versions support only one
encoding per server instance. PostgreSQL supports Unicode and a bunch
of other encodings right now, selectable per database. Unicode is
essential for the planned support of MARC/UNIMARC records.

 > 
 >  - At first, there is no sample files in order to test the local
 > installation. This is a pity.

Granted. RefDB should include a bunch of sample files.

 > For example, my BibTeX file could not be input from the refdb WEB interface
 > for an unknown reason (with a javascript warning saying that it was not a
 > BibTeX file!); and I had liked to see how is a standard BibTeX/refdb input
 > (is it with { or ( etc.?). Also there is no documentation on what refdb does
 > with the "\citekey" field.

bib2ris (the app that does the BibTeX-to-RIS conversion behind the web
interface) uses the btparse library, so anything that this widely used
library can grok will work with RefDB. I personally ran tests with
files according to the Kopka/Daly guide and with random files pulled
from the web.

 > I name citekey the text as in: @article(citekey,
 > 
 > The citekey appears as the ID field of the RIS file after a bib2ris
 > conversion but it disappears in the database and is replaced by a numbered
 > ID
 >  @article(db-1,
 > 
 > It is a severe limitation and truncation of your bibtex file which has to be
 > documented.

Support for user-defined IDs is not yet implemented.

 > 
 >  - The first description and advertisement of RefDB on sourceforge.net
 > compares RefDB with Reference Manager and EndNote; it is far from real. The
 > main two advantages of these commercial products (and Procite too) consist
 > in my view in: (1-import) on the "rapid" selection and "automatic" inclusion
 > in an temporary database of references obtained from a search on Medline (or
 > whatever bibliographic server), (2-export) on the "cite while you type"
 > facility which you can use from Word or Word perfect by querying your
 > reference manager.
 > 
 > As far as I understand refdb today in order to simulate the first feature
 > (1-import) you have to search your Medline with your own browser, copy and
 > paste the different parts of the text of interest into your refdb browser
 > (same browser but another and appropriate window). You can do it also from
 > the command line but it doesn't seem to be more simple.
 > 
 > Feature (2-export) can only be done with LaTeX/BibTeX (I am happy because
 > LaTeX was what I wanted but most of the people here would like to use the
 > refdb database with Word!). In most of these commercial products you can
 > enter some "citekeys" into your text and if you don't remember the exact ID
 > of your reference you can enter an author and/or a keyword and/or a date
 > (which have to be enclosed by braces or something like that) and then by
 > clicking from Word on the "Scan document" button you can select from your
 > reference database one or more exact references to be inserted. Also refdb
 > is not a Z39.50 server and can't be used with such, commercial or not,
 > reference seekers. It has to be documented.
 > 

You should keep in mind is that RefDB was designed for use with
SGML/XML documents. LaTeX support is a pleasant side effect with some
still unresolved issues. So the main goal of development is not to
create a free Reference Manager lookalike but to develop a Unix-style
tool that allows you to arrive at the same result (a printable or HTML
document with a correctly formatted bibliography), but you can use
markup languages instead of proprietary word processors. What you talk
about here are usability issues from the point of view of a Windows
user (I assume). For a Windows user, there is one window that you
interact with, and this one window must have all the capabilities that
you need for a particular job. Unix has a different philosophy. A job
is done by concatenating a bunch of tools that you feel are suited
best for your needs. When I write a DocBook document, I have a few
Emacs windows on one virtual screen and half a dozen xterms on another
virtual screen, and I can do whatever I need in these windows. And if
someone wants to use vi that's ok too. Keep in mind that you basically
don't have a choice with commercial products. You have this close
integration of the bibliography stuff because one commercial product
is tighly tied to another commercial product on one particular
commercial platform. Try clicking on "Scan document" in OpenOffice on
FreeBSD and you see what I mean (there is no such thing). With RefDB
you have the choice to use whatever you like, at the price of
requiring some work to integrate it into your personal environment.

This approach offers the benefit of "scriptability". As an example,
you could subscribe to an email service that runs your favourite
Medline query once a week, mark the references you're interested in,
and hand the data to a script that adds these references to a RefDB
database. Try that with a commercial product.

Taken together, RefDB does not try to simulate the usability features
of commercial Windows products. It uses a different approach to arrive
at the same final result, a formatted bibliography.

Z39.50 support will eventually be implemented, but strictly according
to this philosophy: as an external application.

 >  o Comments and questions on the installation documentation
 > 
 >  - Why during the make install, the style files (bibtex-full.xml etc.) are
 > copied to /usr/local/share/refdb/styles ? It seems that during a:
 >   refdbib -u username -d foo -S bibtex-full -t bibtex foo.aux
 > command, refdbib does not look at that style file directory. Ok, I
 > understand that the styles have to be loaded into the refdb mysql database
 > but why, again, the refdba command:
 >   addstyle bibtex-full.xml
 > does not look at that special directory? The only way I found to add styles
 > was to enter the complete path
 >   addstyle /usr/local/share/refdb/styles/bibtex-full.xml
 > Is there another way? (I spent hours on this!)

No. It is a basic feature of the Unix command line that if you want to
work with a particular file you use its path to specify this
file. RefDB installs the example styles in a useful place, but it does
not force you to add your own styles to this particular directory.

 > What means the line
 >  stylespecdir $REFDBLIB/stylespec
 > in refdbibrc?

Sorry about that. This is leftover junk that needs to be cleaned up.

 > 
 > It is not clear in your README or INSTALL file that after "make install" you
 > should have a refdbd running...
 > ...and refdba works without a refdbd running...
 > I mean that there is no error message if you create a database with refdba
 > without a refdbd running!

I can't reproduce this. If I run any command from refdba or refdbc
without a refdbd instance running I get:

could not establish server connection

which should be pretty descriptive. Please verify that you don't have
a refdbd process running in the background.

 > Many of the refdb client don't have error messages (like refdbib if the
 > foo.aux does not exist).
 > 
 > What is the default style of refdbib?

There is no default style. If you don't specify a particular style,
you'll get an error message.

 > Why isn't the default style not included into the mysql database during the
 > make install, or during the input of a mysql install dump.
 > 
 > In fact you could add also the inclusion of a sample database (foo) whith a
 > dump.
 > 

Yep. Dumps would be good.

 > Instead of long documentation sections on how to implement refdbd daemon
 > which every one knows (/etc/rc.d/init.d/refdbd start) you could simply tell

*You* happen to know that. But you are not everyone. And what you
 suggest happens to work on your favourite Linux distro, but not on
 FreeBSD or Cygwin or a dozen other systems. This is why this stuff is
 an integral part of any comprehensive documentation.

 > people to launch refdbd as:
 >   refdbd -s -V -e 0 -l 7
 > in order to test the connection.
 > 
 > For example the creation of 'root' and other user is not clear. Even if I
 > use mysql for other product like the 'sympa' mailing list server,
 > phpmyAdmin, and other private services I spent a lot of time until it works.
 > 
 > For example, refdb seems to need "localhost.localdomain" as the local host.
 > It is not true with other standard mysql services where simply "localhost"
 > is the host. I have had to create a root user for localhost.localdomain,
 > because the standard one did not work: only an interactive (-e 0) refdbd
 > launch can help you to see the:
 >   Access denied root@localhost.localdomain
 > messages.
 > 

It is frankly stated in the manual that you still need some MySQL
administration skills to set up RefDB (and it is a stated goal for
the first stable release 1.0 to change that...). This includes reading
the excellent MySQL manual. I don't know why you had to create 'root'
as it is the default administrator on fresh MySQL installations. If
you removed this user for security reasons, I can assume you know what
you were doing.

I've never seen the "localhost.localdomain" issue on any of my
installations. It would be great if you could supply more information
about your setup in order to find out what happens. I can use
"localhost" or "127.0.0.1" (if I allow TCP/IP connections) without any
problems on several platforms (FreeBSD, Debian, WinNT).

 > Also, there is no documentation on the port that you use and, if you are
 > protected by a firewall, which one you can use. Is port 9734 planned to be
 > included in /etc/services like 3306 is for mysql?
 > 

This depends on how you set up RefDB. If you run clients outside the
firewall, then you have to allow access to port 9734. If refdbd and
mysqld run on the same box, port 3306 does not have to be accessible
from outside.

 > It seems to me that if you use a WebInterface of refdb the access are done
 > from localhost so you don't need to open the 9734 port for input with your
 > firewall.

If you use the web interface, Apache locally runs refdbc which in turn
connects to refdbd which in turn connects to mysqld. If all of this
happens on the same box, you don't need any publicly accessible port
except 80 for Apache.

 > 
 > Your documentation is mainly written as if clients were all on Unix boxes in
 > a World without firewalls. I think that reality changed. It is not necessary
 > to create users with -H % privileges if your mysql databases are used for
 > other purposes like confidential mailing lists with protected passwords.
 > 

Again, the RefDB manual can't repeat the MySQL manual and your
favourite Firewall's manual. If you use these tools, you should use
the appropriate documentation. The -H % examples are nothing but
examples with the hints that you should use restrictions appropriate
to your local installation.

 > There is no section on "Common Errors" like "What to do if
 > you get Access denied" or other things like that. The use of phpMyAdmin
 > could be mentioned and emphasized as it really simplifies the management of
 > databases.

Granted. The common errors section is badly missing.

 > 
 > Also I don't understand the personal reference list. What is the hierarchy
 > between users? What are the privileges to setup? If someone uploads a
 > bibliography from the Web interface, would it be a common database or a
 > personal database? Also a guided tour could be nice.

There is no such thing as a hierarchy. Personal reference lists are a
tool to *avoid* personal databases. The idea is to pool all references
that a workgroup or a department collects in one database and still allow
each user to have a view of those references that he personally added
(or picked from those that others added), just as if he had used a
private database. This is independent of whether you use the web
interface or the CLI.

 > 
 > Your documentation is not numbered and its structure is not clear.
 > For example in your Table of Contents you have Parts and Chapters but
 > chapters are not renumbered from one after each part. Under each chapter,
 > sections are not numbered at all. It is very confusing with subsections. For
 > example why "Finishing the refdb installation" which is a major section is
 > before the "Seting up the refdb web services". Again "Add or delete
 > bibliography styles" should be partly dupplicated in a sub-section of
 > Finishing the refdb installation if not setup through a mysql command file
 > or dump.

The manual was transformed with the default DocBook DSSSL
stylesheets. The numbering stuff could be altered by a driver file if
you feel this improves readability.

As for the other issues, the manual is certainly far from perfect. 
 > 
 > There is actually no documentation on the Web interface which is what I am
 > interested mostly in. The "login" button is confusing: whatever username and
 > password you enter, the message is that "Your are logged in". What is this
 > button doing? The java script could be included in the HTML to be more
 > clear. I understand that you need a username, a password and a database to
 > submit a query. So the Login button is not mandatory. In your simple query a
 > book title is not searched, only a title of an article is searched.
 > 

Including the Javascript code in the HTML would duplicate the code in
a dozen files. This is why it is in an easily maintainable external
file that you can view with your browser as easily as a HTML page.

The purpose of the login button is to set the cookies for the
authentication stuff. This is necessary if you immediately switch to a
different page without running a query (the latter creates the cookies
on the fly), because all other pages assume that the cookies already
exist.

 > I know that criticism is easy but when I see that the todo list is mostly
 > concerned by new facilities I am afraid that many potential users,
 > sourceforge downloaders, did not try the product only because the
 > documentation is not clear and not because of the limitations of refdb.
 > 
 > Also it is not difficult to describe briefly in the introduction what are
 > the other softwares or concepts that you use (BibTeX, SGML, XML, TEI,
 > DocBook etc.). Even LaTeX is not described at all.

RefDB is a tool for people that have specific requirements to get
something done. If a XML author needs bibliographies, I don't have to
explain XML to him. If a LaTeX author needs bibliographies, I don't
have to explain LaTeX to him. If you don't know it, you most likely
don't use it. I believe what you ask for is beyond the scope of a user
manual. It would be a nice introductory chapter of an O'Reilly book on
RefDB (just kidding).

[...]

 > I hope that my remarks and comments were helpful and not offensive because
 > refdb is probably one of the most "opensource" innovative product for
 > scientists I have found.

Again, I appreciate your comprehensive comments. Still you should keep
one thing in mind: RefDB is essentially a one-man show (but thanks to
all who sent in bug reports, did testing on arcane platforms, and sent
in Makefile patches etc). I'm hacking RefDB in my spare time according
to my personal needs, but I always try to listen what others would
like to see implemented. But you simply can't expect a free Reference
Manager clone from me for all platforms that Reference Manager ignores
and for all document types that Reference Manager ignores with all the
nifty usability features of a $500 package. (And now the free software
mantra:) RefDB is free software. If you want to improve RefDB, take
the source, implement missing features, and send in the patches. If
you want a better documentation, take the DocBook sources, improve the
document, and send in the patches. I hope this doesn't sound rude but
this is how most non-commercial software packages are developed these
days.

best regards,
Markus

-- 
Markus Hoenicka <hoe...@co...>
http://ourworld.compuserve.com/homepages/hoenicka_markus/