refdb-users — List for RefDB users

[Refdb-users] ANN: RefDB-0.9.3-pre3 available

[Markus H. <mar...@mh...>]

Hi all,

there's a new pre-release of the upcoming 0.9.3 along with an updated
Perlmod package available right here:

http://refdb.sourceforge/net/pre/refdb-0.9.3-pre3.tar.gz
http://refdb.sourceforge/net/pre/RefDB-perlmod-0.3_pre1.tar.gz

There's still no risx import, that's why it's just a pre-release. The
following issues have been improved or fixed compared to 0.9.2a:

- configure.in fixes for OSX

- contains risx.dtd

- new backend for risx export (use getref -t risx)

- refdbc running as CGI app no longer checks for username/password
  before reading CGI data

- fixed several bugs related to automatic generation of citation keys

- fixed problem with detection of duplicate journal entries

- fixed problem with web interface in Windoze browsers

- docbook and tei backends now support getref -s switch to request
  additional tags, use -s ALL to retrieve all suitable fields. Both
  now support notes and abstracts to pretty-print annotated
  bibliographies

- getref queries including :JO: or one of the other journal types now
  perform a regexp match instead of a full match

- fixed horrible get_periodical bug that corrupted memory

- fixed missing quotes in docbook author output

- new import filter en2ris.pl to import EndNote "RIS" data

- med2ris.pl now uses iconv to do character encoding conversion on the
  fly

Comments and suggestions are welcome.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Bruce D'Arcus writes:
 > 
 > On Wednesday, April 30, 2003, at 12:54  PM, Markus Hoenicka wrote:
 > 
 > > Oops. Please read this as:
 > >
 > > Y2  - ///December 1 OtherInfo
 > > Y2  - ///Dec 1 OtherInfo
 > >
 > > according to the sample data Bruce sent. The slashes are optional so 
 > > the
 > > version above would work as well.
 > 
 > OK, so is it better to include the slashes in the export filter, or 
 > not?  Once I finish it, I'll send it to you and you can host it if you 
 > want.
 > 

They're basically not required, so you can leave them out if you
wish. I'll be happy to host the export filter once it's done.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

On Wednesday, April 30, 2003, at 12:54  PM, Markus Hoenicka wrote:

> Oops. Please read this as:
>
> Y2  - ///December 1 OtherInfo
> Y2  - ///Dec 1 OtherInfo
>
> according to the sample data Bruce sent. The slashes are optional so 
> the
> version above would work as well.

OK, so is it better to include the slashes in the export filter, or 
not?  Once I finish it, I'll send it to you and you can host it if you 
want.

Bruce

Re: [Refdb-users] using refdb with a course website

[Markus H. <mar...@mh...>]

Hi Matt,

Matt Price writes:
 > are the existing backends well-documented?  I haven't had a chance to
 > poke around in the source yet, but maybe if you can point me to the
 > main locations I can try to learn enough about them to figure out
 > where to start.  
 > 

Sorry, no. You'll have to wade through C code but that contains quite
some comments. The places to look at are the backend_xyz.c files, each
of which defines a backend (except the bibliography backend which is
split in two due to its complexity). There's essentially three functions being
called by getref, one that adds a header (if required), one that
formats the data proper, and one that adds a footer (if
required). backend.c defines the internal API which is the most
convenient way to get at the individual fields.
 
 > and maybe I can learn perl while I'm at it...  
 > 

The more I think about it, the more I'm convinced that custom backends
are best implemented on top of the Perl client module, using risx as
the data exchange format. So there's indeed a good reason to learn
Perl.

 > I think risx is probably the answer for me, unless DcBook is so
 > enormously powerful that it could generate files that could be easily
 > read by EndNote (I don't know much about docBook, but it does seem
 > very cool).  The target audience here is students and researchers in
 > the humanities -- that is, people even dumber than I am about
 > technical questions.  
 > 

?? If you're thinking about getting RefDB data into EndNote, then
plain ol' RIS should be the answer. RIS as exported by RefDB may need
some tweaking to get it into EndNote cleanly (as is the case if you go
the other way), but I would certainly look no further for this task.

 > OK, I'll look at the module.  But suppose (in the interim, while I'm
 > learning) I set up a database with the same structure as refdb's and
 > enter references from the web, or via FileMaker or something, using
 > only the main table.  Will it be straightforward to add these
 > references to RefDB at a later date?  
 > 

Keep in mind that the main table lacks the journal, the author/editor,
and the keyword information, all of which I consider essential.

 > And on a similar note:  I would *love* to have the refdb tables in
 > the same database as that which is maintained by SPIP, the CMS I'm
 > working with.  This would allow me to make cross-referencing tables
 > for SPIP to use when it generates static html pages for web display.
 > I suppose this kind of thing could be done even if I were using
 > separate databases, but I'd rather not have that hurdle if I can help
 > it.  Is there any in principle problem with this?  I noticed that
 > refdb wouldn't create the necessary tables in a pre-existing
 > database, and wondered if I could work around that.  
 > 

Please look at the empty.mysql.dump and empty.pgsql.dump.in scripts
shipped with RefDB. The main use of these scripts is to allow database
administrators that don't want to run an app they don't know to create
a RefDB database. However, if you run these scripts on existing
databases, they will create the RefDB-specific tables unless there are
name clashes. empty.mysql.dump should work out of the box, whereas
empty.pgsql.dump.in needs some modification as it attempts to create
the database itself. Just remove the offending code and off you go.

 > well, I'm NOOOOOTTTT a hacker, honestly, but I will try!

Great!

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Hi Bruce,

I'm not too optimistic that 0.9.3 will be out anytime soon. This is an
overview of what I intend to get fixed for the next release:

risx import (export seems to work): requires a fairly extensive
refactoring and modularization of the data import routines (the good
thing is: once this is done, other native formats could be added
easily).

risx documentation: the key to get people use it

getref -s switch: should work in DocBook and TEI backends too

manual keyword scan: currently only the automatic keyword scan works
but this necessarily leaves out the keywords that are added later.

use Text::Iconv in med2ris.pl: while we're at it...

DocBook backend: reportedly has stability problems with some data,
although I believe I've fixed the underlying problem (thanks Marc!!)

reference permission problem: Who may delete datasets, if at all?

I'll try to get as much as possible done tomorrow (Labor day over here
in Germany :-), but I guess it'll take weeks until the code is
complete and more or less tested.

I plan to make available a snapshot of the current CVS version
(0.9.3-pre3 I guess) as this includes lots of fixes without too many
untested features that could get into your way.

regards,
Markus

Bruce D'Arcus writes:
 > Question: when do you expect to release 0.9.3, and what changes if any 
 > will be added at this point compared to the current CVS version?  I 
 > want to get this all taken care of by end of next week, but am not 
 > going to have much time for a few days anyway...
 > 

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Markus Hoenicka writes:
 > This version can deal with dates that look like (assuming PY has the
 > year):
 > 
 > Y2  - December 1 OtherInfo
 > Y2  - Dec 1 OtherInfo
 > 

Oops. Please read this as:

Y2  - ///December 1 OtherInfo
Y2  - ///Dec 1 OtherInfo

according to the sample data Bruce sent. The slashes are optional so the
version above would work as well.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

On Tuesday, April 29, 2003, at 06:16  PM, Markus Hoenicka wrote:

> I've checked in an extended version of the en2ris Perl script (it's
> called en2ris.pl.in as it is modified during make).

Thanks Markus!

Question: when do you expect to release 0.9.3, and what changes if any 
will be added at this point compared to the current CVS version?  I 
want to get this all taken care of by end of next week, but am not 
going to have much time for a few days anyway...

Bruce

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Hi,

I've checked in an extended version of the en2ris Perl script (it's
called en2ris.pl.in as it is modified during make). This version has
the usual config file and log file stuff that my other Perl scripts
use and therefore require RefDB::Log and RefDB::Prefs from the
refdb-perlmod package. It also offers command-line option handling and
so on. Try "en2ris.pl -h" after building from the CVS version. If you
don't want to rebuild RefDB from the CVS sources, download only
en2ris.pl.in, rename it to en2ris.pl, and replace the string
"<sysconfdir>" with whatever you use on your system
("/usr/local/etc/refdb" is a good guess).

This version can deal with dates that look like (assuming PY has the
year):

Y2  - December 1 OtherInfo
Y2  - Dec 1 OtherInfo

Both the day and the OtherInfo are optional.

regards,
Markus

Bruce D'Arcus writes:
 > Unfortunately, since the field is just dumb text, they can be anything 
 > the user chooses.  For my records, they are indeed consistent.  I'm not 
 > sure about other Endnote users, but I expect English-language records 
 > are likely to use my approach or the abbreviation.
 > 

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

On Monday, April 28, 2003, at 07:08  PM, Markus Hoenicka wrote:

> On the dates: Are they predictable, sort of? That is, can I always
> expect to find something like "December 1" in Y2 or could it as well
> be "Dec 1" or whatever? I could do a translation if these dates show
> up in a consistent format, or at least a small number of discernible
> formats.

Unfortunately, since the field is just dumb text, they can be anything 
the user chooses.  For my records, they are indeed consistent.  I'm not 
sure about other Endnote users, but I expect English-language records 
are likely to use my approach or the abbreviation.

Bruce

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Hi Bruce,

the current script expects both the SP and EP tags, so please leave
them in (it's a tad easier to deal with for me).

On the dates: Are they predictable, sort of? That is, can I always
expect to find something like "December 1" in Y2 or could it as well
be "Dec 1" or whatever? I could do a translation if these dates show
up in a consistent format, or at least a small number of discernible
formats.

Assuming that Y2 may contain random stuff, the script attached below
should work now (I'll put it in CVS shortly, and it'll have nice
command-line parameters to select the encoding, and it'll read a
config file and such, eventually). It requires Text::Iconv from your
favourite CPAN mirror. I run it here like this, using your Mac-style
test file:

tr "\r" "\n" < endnote_export.ris | en2ris.pl > out.ris

On a Mac you may have to run it like this instead:

en2ris.pl < endnote_export.ris | tr "\r" "\n" > out.ris


regards,
Markus

Bruce D'Arcus writes:
 > 
 > On Monday, April 28, 2003, at 04:33  PM, Markus Hoenicka wrote:
 > 
 > > en2ris.pl fiddles only with the tags (I forgot to take care of the
 > > dates, though).
 > 
 > And on the dates, will my suggested solution work for the export 
 > filter?  I will plan to keep (and extend) the SP and EP pages thing....
 > 
 > Bruce

en2ris.pl
--8<------
#!/usr/bin/perl

use Text::Iconv;

my $last_tag = "TY  - ";
my $PY = "";
my $Y2 = "";

my $converter = Text::Iconv->new("MacRoman", "UTF-8");

while (<>) {
    if ($_ =~ /^(..  - )/) {
	$last_tag = $1;
    }
    elsif ($last_tag eq "KW  - ") {
	print $last_tag . $_;
	$_ = "";
    }

    if ($_ =~ /^SP  - /) {
	$_ =~ s/^SP  - (.*)-.*/SP  - $1/;
    }
    elsif ($_ =~ /^EP  - /) {
	if ($_ =~ /^EP  - .*-.*/) {
	    $_ =~ s/^EP  - .*-(.*)/EP  - $1/;
	}
	else {
	    $_ = "";
	}
    }
    elsif ($_ =~ /^PY  - /) {
	chomp $_;
	$PY = substr($_, 6);
	$_ = "";
    }
    elsif ($_ =~ /^Y2  - /) {
	chomp $_;
	$Y2 = substr($_, 6);
	$_ = "";
    }
    elsif ($_ =~ /^ER  - /) {
	# dump pubyear string
	print $converter->convert("PY  - $PY$Y2\n");
	$PY = "";
	$Y2 = "";
    }

    print $converter->convert("$_");
}
--8<------

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

On Monday, April 28, 2003, at 04:33  PM, Markus Hoenicka wrote:

> en2ris.pl fiddles only with the tags (I forgot to take care of the
> dates, though).

And on the dates, will my suggested solution work for the export 
filter?  I will plan to keep (and extend) the SP and EP pages thing....

Bruce

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Hi,

Marc Baaden writes:
 > - books and ser entries are sparely handled (eg no publisher, city
 >   or editors) by the output filter. Needed to add that info in by
 >   hand :(
 >   Actually it would be simpler to export rather a bit too much, as
 >   removing tags from an xml file is not too difficult with a simple
 >   perl script (did that this weekend)

You're right about that. Attribute this to my laziness. This issue has
come up just recently, and I put on my todo list to treat the DocBook
(and TEI, if possible) output in the same way as e.g. the screen
output. Whenever it creates legal markup you'll be able to request
specific fields with the -s switch.

Would you mind sending me a couple of sample datasets by private mail?
I'm mostly working with journal articles so I rarely run into these
problems.

 > 
 > - it wasn't possible to retrieve the references in a given order ..
 >   (.. so I retrieved them one by one in the order I needed and mangled
 >    the xml files with very badly written scripts .. but it worked in
 >    the end)
 > 

getref currently sorts the output either by ID or by PY by using the
-S switch. Fancy sorting is only implemented in the bibliography
backend.

 > - I also first had tried to select a subset of references, using pickref,
 >   but that didn't seem to work/do anything other than throw
 > 
 > duplicate ID 1 ignored
 > 0 datasets added to personal interest list, 1 ignored
 > 
 >   at me ..
 > 

This is most likely a misunderstanding. If you added these datasets
yourself, they're in your personal reference list anyway. Running
pickref again won't do any good.

 >   Is there a prefered way to retrieve a given selection of references from
 >   a database ? (And even better, in a given order ?)
 > 

I'm not sure whether I grasp what you want to do, but you can cobble
together your query using the screen output until you get all
references you need. Then rerun this query with the -t db31 switch to
save the results to a file.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

[Refdb-users] share references

[Markus H. <mar...@mh...>]

Hi,

you've put your finger onto one of the weak spots. Not that it is
technically difficult, but I couldn't decide what to do so this
problem has been dangling essentially since the first release.

In general, removing references from a shared reference database is no
good idea anyway, so if everyone restricts himself to pickref -r no
harm should be done. There's a dark secret related to removing
references: It may cause trouble if you have to re-create the database
from a SQL dump. This problem will be fixed only once the risx
export/import is fully implemented.

If you positively want to remove references, there's two ways for me
to handle it: either treat the person who first added the reference as
the "owner" and let him remove it whenever he feels like it. This
might rightfully anger others that want this dataset. So the right of
the "owner" to delete a dataset could be restricted to datasets that
no one else picked.

OTOH deleting references could be restricted to the database admin
altogether. He could have the right to remove references that no one
picked (that is, when also the "owner" ran pickref -r on them).

Input about how to deal with this will be greatly appreciated.

regards,
Markus

Thomas B. writes:
 > Hi 
 > 
 > Got a little problem with RefDB. If you want to share references on the
 > same database 
 > it is a small thing which disturbs me. If you add a reference everybody
 > else can delete 
 > or change it. I think it is not so good. Can anybody tell me how to
 > change the code that
 > only the person who added the reference can change it.
 > I think it could me made with few modifications but I don't know exactly
 > what to change.
 > Would be nice if anybody can help me
 > thanks
 > Thomas
 > 
 > 
 > -------------------------------------------------------
 > This sf.net email is sponsored by:ThinkGeek
 > Welcome to geek heaven.
 > http://thinkgeek.com/sf
 > _______________________________________________
 > Refdb-users mailing list
 > Ref...@li...
 > https://lists.sourceforge.net/lists/listinfo/refdb-users
 > 

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Hi Bruce,

apologies for being unclear about what I was doing. Let's look at it
again:

en2ris.pl < endnote.ris | iconv -f MacRoman -t UTF-8 | tr "\r" "\n" > out.ris

en2ris.pl fiddles only with the tags (I forgot to take care of the
dates, though). It should not change the encoding or the line endings,
although I'm not sure about this on OSX. The script may fail
altogether if Perl can't deal with the Mac line endings.

the iconv call changes the encoding from MacRoman (apparently what
EndNote on OSX emits) to UTF-8 which we all like better.

The tr call should translate the Mac-style carriage returns to
Unix-style linefeeds. However, Perl might have done that already, I
haven't tested on OSX yet.

My preferred way to find out about line endings is (who'd have
guessed?) Emacs. Just put the following in your .emacs:

;; modify modeline
(setq eol-mnemonic-dos (string-to-char "\\"))
(setq eol-mnemonic-unix (string-to-char "/"))
(setq eol-mnemonic-mac (string-to-char ":"))
(setq eol-mnemonic-undecided (string-to-char "?"))

The defaults are too confusing. When using this code, Emacs will
display the line endings on the left-hand side of the modeline: A
backslash for DOS/Windows files, a colon for Mac files, a slash for
Unix files.

BTW I've finally got Text::Iconv going on my box, so future versions
of en2ris.pl will be able to do the encoding conversion without
running the stuff through iconv.

regards,
Markus

Bruce D'Arcus writes:
 > 
 > I wrote:
 > 
 > > OK, yes, it seems to work (though am not sure how to check the 
 > > line-endings).
 > 
 > Sorry, but correction:
 > 
 > Looking at the script, it seems you were trying to correct the tag 
 > problems too.  This part did not work.  I was referring to the encoding 
 > conversion, which did.
 > 
 > Bruce
 > 
 > 
 > 
 > -------------------------------------------------------
 > This sf.net email is sponsored by:ThinkGeek
 > Welcome to geek heaven.
 > http://thinkgeek.com/sf
 > _______________________________________________
 > Refdb-users mailing list
 > Ref...@li...
 > https://lists.sourceforge.net/lists/listinfo/refdb-users
 > 

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

[Refdb-users] share references

[Thomas B. <tkb...@fs...>]

Hi 

Got a little problem with RefDB. If you want to share references on the
same database 
it is a small thing which disturbs me. If you add a reference everybody
else can delete 
or change it. I think it is not so good. Can anybody tell me how to
change the code that
only the person who added the reference can change it.
I think it could me made with few modifications but I don't know exactly
what to change.
Would be nice if anybody can help me
thanks
Thomas

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

I wrote:

> OK, yes, it seems to work (though am not sure how to check the 
> line-endings).

Sorry, but correction:

Looking at the script, it seems you were trying to correct the tag 
problems too.  This part did not work.  I was referring to the encoding 
conversion, which did.

Bruce

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

On Sunday, April 27, 2003, at 03:30  PM, Markus Hoenicka wrote:

> Please find a real simple Perl script attached that converts the UTF-8
> version of your data to something useful.

OK, yes, it seems to work (though am not sure how to check the 
line-endings).  The only unfortunate thing is that iconv doesn't seem 
to be installed on OS X by default.  There's a Fink package for it 
though, so no big deal.

It would be really nice to see everything up to (and including at some 
point) be handled by Fink, in particular libdbi and its drivers.

Bruce

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Bruce D'A. <bd...@fa...>]

On Sunday, April 27, 2003, at 03:30  PM, Markus Hoenicka wrote:

> Please find a real simple Perl script attached that converts the UTF-8
> version of your data to something useful.

You mean "MacRoman"?  By default Endnote saves to MacRoman on the Mac, 
and Latin-1 is unsuitable because it doesn't support some characters 
that I tend to use a lot (like curly quotes!).  So ideally I'd like 
RefDB to convert from MacRoman to utf and send it to, in my case, 
PostgreSQL.

This raises the question: is it possible for the import system to do 
this?  For example, a Mac user saves to an RIS file, and wants to 
import that file into RefDB that is using a unicode enabled version of 
SQLite.  Or someone gets a utf-8 encoded risX file, but wants to import 
into MySQL and store in Latin-1.  Possible for the import system to 
handle this?

Bruce

Re: [Refdb-users] Endnote RIS export filter (help?)

[Bruce D'A. <bd...@fa...>]

On Sunday, April 27, 2003, at 04:43  PM, Bruce D'Arcus wrote:

> ==========
> GENERIC
> `TY  - `CHAP|`

Well, the first one's wrong!  The above obviously ought to be GEN.

Bruce

[Refdb-users] Endnote RIS export filter (help?)

[Bruce D'A. <bd...@fa...>]

OK, I've managed to clean up a lot of the Endnote RIS export filter, 
but I'd like feedback on whether this is correct.  Below is the 
template to map Endnote field names to RIS tags.  I am most unsure 
about the non-standard things (like the law ref types).  I'll make the 
final export filter available when it's done, and this can be the basis 
for the risX export filter as well (though am not sure there's much 
point).

A couple of things to note:

1)  I have started to repeat the "pages" field in both SP and EP, 
thinking a Perl script can clean this up.  However, I'm now thinking it 
might be better to just put pages in SP and leave out the EP tags.  
That will cover single-page records, and a Perl script can still manage.

2)  I don't know how to deal with dates.  Endnote has no understanding 
of numeric dates other than year, which is a separate field.  As a 
result, I am thinking to take a date like "December 1" and create:

Y2  - \\\December 1

Bruce

==========
GENERIC
`TY  - `CHAP|`
AU  - `Author|`
PY  - `Year|`
T2  - `Secondary Title|`
A2  - `Secondary Author|`
T1  - `Title|`
CY  - `Place Published|`
PB  - `Publisher|`
T3  - `Tertiary Title|`
A3  - `Series Editor|`
ET  - `Edition|`
SP  - `Pages|`
EP  - `Pages|`
Y2  - `Date|`
SN  - `ISBN/ISSN|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL|`
U1  - `Custom 1|`
U2  - `Custom 2|`
U3  - `Custom 3|`
U4  - `Custom 4|`
U5  - `Custom 5

JOURNAL ARTICLE
`TY  - `JOUR|`
AU  - `Author|`
PY  - `Year|`
TI  - `Title|`
SP  - `Pages|`
JF  - `Journal|`
JO  - `Alternate Journal|`
VL  - `Volume|`
IS  - `Issue|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

BOOK
`TY  - `BOOK|`
AU  - `Author|`
PY  - `Year|`
T1  - `Title|`
CY  - `City|`
PB  - `Publisher|`
ET  - `Edition|`
VL  - `Volume|`
Y2  - `Original Publication|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL

BOOK SECTION
`TY  - `CHAP|`
AU  - `Author|`
PY  - `Year|`
T2  - `Book Title|`
A2  - `Editor|`
T1  - `Title|`
CY  - `City|`
PB  - `Publisher|`
ET  - `Edition|`
VL  - `Volume|`
T3  - `Series Title|`
SP  - `Pages|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL

MANUSCRIPT
`TY  - `PAMP|`
AU  - `Author|`
PY  - `Year|`
BT  - `Collection Title|`
T1  - `Title|`
CY  - `City|`
SP  - `Pages|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
VL  - `Number|`
AD  - `Author Address|`
UR  - `URL

EDITED BOOK
`TY  - `BOOK|`
A2  - `Editor|`
PY  - `Year|`
T2  - `Series Title|`
ED  - `Series Editor|`
T1  - `Title|`
CY  - `City|`
PB  - `Publisher|`
KW  - `Keywords|`
N1  - `Notes|`
N2  - `Abstract|`
ET  - `Edition|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL

MAGAZINE ARTICLE
`TY  - `MGZN|`
AU  - `Author|`
PY  - `Year|`
T2  - `Magazine|`
T1  - `Title|`
SP  - `Pages|`
EP  - `Pages|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL

NEWSPAPER ARTICLE
`TY  - `NEWS|`
AU  - `Reporter|`
PY  - `Year|`
T2  - `Newspaper|`
T1  - `Title|`
CY  - `City|`
ET  - `Edition|`
SP  - `Pages|`
VL  - `Volume|`
Y2  - `Issue Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
ET  - `Edition|`
AD  - `Author Address|`
UR  - `URL

CONFERENCE PROCEEDINGS
`TY  - `CONF|`
AU  - `Author|`
PY  - `Year of Conference|`
T2  - `Conference Name|`
ED  - `Editor|`
T1  - `Title|`
CY  - `Conference Location|`
PB  - `Publisher|`
T3  - `Series Title|`
ED  - `Series Editor|`
ET  - `Edition|`
SP  - `Pages|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
ET  - `Edition|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL

THESIS
`TY  - `THES|`
AU  - `Author|`
PY  - `Year|`
T2  - `Academic Department|`
T1  - `Title|`
CY  - `City|`
PB  - `University|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

REPORT
`TY  - `RPRT|`
AU  - `Author|`
PY  - `Year|`
CT  - `Title|`
CY  - `City|`
PB  - `Publisher|`
T3  - `Tertiary Title|`
SP  - `Pages|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

PERSONAL COMMUNICATION
`TY  - `PCOMM|`
AU  - `Author|`
PY  - `Year|`
T1  - `Title|`
A2  - `Recipient|`
CY  - `City|`
PB  - `Publisher|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

COMPUTER PROGRAM
`TY  - `COMP|`
AU  - `Programmer|`
PY  - `Year|`
CT  - `Title|`
CY  - `City|`
T3  - `Tertiary Title|`
SN  - `ISBN/ISSN|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

ELECTRONIC SOURCE
`TY  - `ELEC|`
AU  - `Author|`
PY  - `Year|`
CT  - `Title|`
T3  - `Tertiary Title|`
Y2  - `Last Update Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

AV MATERIAL
`TY  - `VIDEO|`
AU  - `Author|`
PY  - `Year|`
BT  - `Collection Title|`
CT  - `Title|`
CY  - `City|`
T3  - `Tertiary Title|`
Y2  - `Date|`
N1  - `Notes|`
N1  - `Extent of Work|`
N1  - `Performers|`
N1  - `Short Title|`
KW  - `Keywords|`
N2  - `Abstract|`
ET  - `Number|`
AD  - `Author Address|`
UR  - `URL

FILM OR BROADCAST
`TY  - `MPCT|`
AU  - `Writer|`
PY  - `Year Released|`
BT  - `Series Title|`
CT  - `Title|`
CY  - `City|`
T3  - `Tertiary Title|`
Y2  - `Date Released|`
Y2  - `Original Publication|`
SN  - `ISBN|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

ARTWORK
`TY  - `ART|`
AU  - `Artist|`
PY  - `Year|`
CT  - `Title|`
CY  - `City|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
ET  - `Edition|`
VL  - `Volume|`
AD  - `Author Address|`
UR  - `URL|`
U1  - `Custom 1|`
U2  - `Custom 2|`
U3  - `Custom 3|`
U4  - `Custom 4|`
U5  - `Custom 5

MAP
`TY  - `MAP|`
AU  - `Cartographer|`
PY  - `Year|`
CT  - `Title|`
CY  - `City|`
PB  - `Publisher|`
T3  - `Tertiary Title|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL|`
U1  - `Custom 1|`
U2  - `Custom 2|`
U3  - `Custom 3|`
U4  - `Custom 4|`
U5  - `Custom 5

PATENT
`TY  - `PAT|`
AU  - `Author|`
PY  - `Year|`
BT  - `Published Source|`
CT  - `Title|`
CY  - `Country|`
SP  - `Pages|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
AD  - `Author Address|`
UR  - `URL

HEARING
`TY  - `HEAR|`
PY  - `Year|`
BT  - `Committee|`
CT  - `Title|`
CY  - `City|`
T3  - `Legislative Body|`
SP  - `Pages|`
ET  - `Session|`
Y2  - `Date|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
UR  - `URL

BILL
`TY  - `BILL|`
PY  - `Year|`
BT  - `Code|`
CT  - `Title|`
T3  - `Tertiary Title|`
SP  - `Code Pages|`
Y2  - `Date|`
SN  - `ISBN/ISSN|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
ET  - `Session|`
VL  - `Code Section|`
UR  - `URL

STATUTE
`TY  - `STAT|`
PY  - `Year|`
BT  - `Code|`
CT  - `Name of Act|`
SP  - `Pages|`
Y2  - `Date Enacted|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
ET  - `Session|`
VL  - `Code Number|`
UR  - `URL

CASE
`TY  - `CASE|`
PY  - `Year|`
BT  - `Reporter|`
CT  - `Case Name|`
SP  - `First Page|`
Y2  - `Date Decided|`
N1  - `Notes|`
KW  - `Keywords|`
N2  - `Abstract|`
VL  - `Reporter Volume|`
UR  - `URL

FIGURE
`TY  - `GEN|`
AU  - `Created By|`
PY  - `Year|`
BT  - `Title|`
N1  - `Image Source Program|`
Y2  - `Date|`
KW  - `Keywords|`
N1  - `Notes|`
N1  - `Image|`
N1  - `Caption

CHART
`TY  - `GEN|`
AU  - `Created By|`
PY  - `Year|`
BT  - `Title|`
N1  - `Image Source Program|`
Y2  - `Date|`
KW  - `Keywords|`
N1  - `Accession Number`
N1  - `Notes|`
N1  - `Image|`
N1  - `Caption

EQUATION
`TY  - `GEN|`
AU  - `Created By|`
PY  - `Year|`
BT  - `Title|`
N1  - `Image Source Program|`
Y2  - `Date|`
KW  - `Keywords|`
N1  - `Accession Number`
N1  - `Notes|`
N1  - `Image|`
N1  - `Caption

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Hi Bruce,

Bruce D'Arcus writes:
 > 2)  Speaking of minor issues, on pages, I'm not sure my solution is 
 > ideal.  It might be better to just export the field once to SP, and 
 > then have a Perl script look for the dash and break up the page range 
 > then into SP and EP.  Reason: what happens if the "pages" is just a 
 > single page?  What do you suggest Markus?
 > 

This is no big deal. It is a little simpler to drop the EP line if it
is no range than to split SP into two lines if it is a range.

Please find a real simple Perl script attached that converts the UTF-8
version of your data to something useful. I'm far away from my Perl
book, so even if it works it may be embarrassing code. I'd like to
gradually extend this until it works for most of your and Marc's
data.

A nice feature would be automatic conversion of line endings (I need
my book for this) and character encoding conversion (this would
require Text::Iconv to work which doesn't work on my system). I'll
look into that later. Currently, something like the following should
work on OSX, assuming that Perl uses the native Mac line endings
(untested). Other platforms can skip the line endings conversion:

en2ris.pl < endnote.ris | iconv -f MacRoman -t UTF-8 | tr "\r" "\n" > out.ris

regards,
Markus

en2ris.pl
--8<-----------------
#!/usr/bin/perl

my $last_tag = "TY  - ";

while (<>) {
    if ($_ =~ /^(..  - )/) {
	$last_tag = $1;
    }
    elsif ($last_tag eq "KW  - ") {
	print $last_tag . $_;
	$_ = "";
    }

    if ($_ =~ /^SP  - /) {
	$_ =~ s/^SP  - (.*)-.*/SP  - $1/;
    }
    
    if ($_ =~ /^EP  - /) {
	if ($_ =~ /^EP  - .*-.*/) {
	    $_ =~ s/^EP  - .*-(.*)/EP  - $1/;
	}
	else {
	    $_ = "";
	}
    }

    print $_;
}
--8<-----------------

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Problems with XML docbook and RefDB bibliographies

[Markus H. <mar...@mh...>]

Hi,

Marc Baaden writes:
 > bib.xml:270: error: AttValue: " or ' expected
 > <author><firstname>C</firstname><othername role=mi>U</othername><surname>Press<
 > 
 > e.g. missing quotes around mi (should be role="mi"), which I can change
 > manually. I'd be also glad to correct it in the corresponding stylesheet,
 > but I haven't quite yet grasped refdb's logic as to which driver file is
 > used for this conversion .. could you comment ?
 > 

There's not much logic involved in this :-). The bug is in backend-db31.c which
creates the markup. I've fixed this in CVS. Thanks for pointing this out.

 > Are there any options/parameters to the driver stylesheet ? Eg I am not
 > interested in details like isbn, issuenum etc. So I'd just like a selection
 > of items/turning some off.
 > 

I'd have to fix this in backend-db31.c too. As a short-term fix you
could also use a simple stylesheet that extracts from the DocBook
output what you need.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Problems with XML docbook and RefDB bibliographies

[Marc B. <ba...@sm...>]

Hi again,

mar...@mh... said:
>> What you're basically asking for is a software that creates plain
>> DocBook output without the formatting overhead. You can do this with
>> RefDB by manually creating bibliographies. Simply run a query and
>> request DocBook output. This will be plain DocBook markup without the
>> overhead required for the formatting. These bibliography elements
>> should work fine with the standard stylesheets. 

Yes indeed. I am experimenting with this.
A small problem is:

bib.xml:270: error: AttValue: " or ' expected
<author><firstname>C</firstname><othername role=mi>U</othername><surname>Press<

e.g. missing quotes around mi (should be role="mi"), which I can change
manually. I'd be also glad to correct it in the corresponding stylesheet,
but I haven't quite yet grasped refdb's logic as to which driver file is
used for this conversion .. could you comment ?

Are there any options/parameters to the driver stylesheet ? Eg I am not
interested in details like isbn, issuenum etc. So I'd just like a selection
of items/turning some off.

Thanks in advance,
  Marc Baaden


-- 
 Dr. Marc Baaden  - Institut de Biologie Physico-Chimique, Paris
 mailto:ba...@sm...      -      http://www.marc-baaden.de
 FAX: +49 697912 39550  -  Tel: +33 15841 5176 ou +33 609 843217

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Markus H. <mar...@mh...>]

Marc Baaden writes:
 > A priori I am using the CVS version which I had updated last weekend.
 > 

Fine. Your data don't trigger the citation key bugs anyway, so
upgrading would not have helped.

It is quite apparent that your data show the same problems that Bruce
reported previously. EndNote's RIS export is lousy, there's no use
denying it. Bruce has put in some effort to fix this, and I recommend
to take a close look at what he came up with (I haven't tested it yet,
I don't have EndNote anyway).

I've fed your data to my local testbed, using:
PostgreSQL 7.2.1 on i386-portbld-freebsd4.3, compiled by GCC 2.95.3
as the database server. Lets have a look at the errors for
each dataset:

Set 0:
For some reason, the abstract ended up as JF (journal, full
name). postgres refuses to accept the value as it exceeds the length
limit for the JF field (255).

Sets 1,2:
This time the abstract was exported as A3 (series editor). postgres
grumbles for the very same reason.

Set 4:
Works for me.

Set 5:
The keywords are not properly tagged. From refdbd's point of view, the
whole thing is a single keyword which again exceeds the length limit,
so postgres rightfully rejects it.

Set 6,7,8:
Work for me.

Set 9:
The address information ends up in the IS (issue or chapter number)
field, once again exceeding the length limit for that field.

Sets 10,11,12:
Work for me

"Work for me" in this case means that refdbd doesn't complain when
importing them. However, some of the datasets cannot be retrieved
afterwards as they lack information which is considered essential for
the given reference type (refdbd should either fill in defaults or
reject the datasets with a proper error message. I'll look into that).


 > Yep. I am running RefDB locally on a machine Dual PIII-700 with 512 MB
 > RAM using a postgres database on the same machine. So as it' s on the
 > same machine, it should not go through the network, shouldn't it ?
 > During the time I checked with top from time to time, and postgres
 > postmaster took between 40 and 90 percent.

There is network traffic between refdbc and refdbd and between refdbd
and postmaster through the loopback device, but this shouldn't hurt
much. I was asking because I've seen slow networks slow down RefDB
when the database server runs on a remote box.

postgres sure is no speed demon, but it has other strengths. MySQL and
SQLite are supposed to be faster, although I don't have any numbers
handy. Could you provide some details how you precisely add the data?
Do you receive the server output on a terminal or do you send the
output to a file (recommended when adding large numbers of
references)? What log level do you use (refdbd -s -e 0 -l 7 slows down
a *lot*)?

 > Endnote swallows the data instantly (let' s say 5 secs)... :(
 > 

I can't believe that. 4000 RIS datasets in 5 secs? Reference Manager
took a couple of seconds per dataset! 

 > 
 > bd...@fa... said:
 > >> I've found the same.  Perhaps the two of us and Markus can figure out
 > >> a  solution? 
 > 
 > Yes, I'd love to.
 > 

I'll see what I can do. I'm not a Perl guru, but maybe my basic
understanding is sufficient to get Bruce's modified output into
shape. Could one of you guys provide me a set of sample data with the
latest EndNote export tweaks? I'd like to see what else needs to be
fixed.


regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-users] Endnote to RIS to RefDB troubles

[Marc B. <ma...@sm...>]

mar...@mh... said:
>> I'll have a look at the ris file. I've experienced a few problems with
>> the 0.9.2a release myself, all of them were related to bugs in the
>> auto-creation of citation keys (the bugs are described on the refdb
>> project page at sourceforge). These bugs have been addressed in the
>> CVS version. I'll find out whether upgrading would help in your case. 

A priori I am using the CVS version which I had updated last weekend.


mar...@mh... said:
>> Could you provide some details about your setup? Which database
>> server, what computer, do you use RefDB across a network etc? I don't
>> have any sort of "benchmark" values available, but an hour sounds like
>> a lot to me. How long would EndNote take to swallow the data? 

Yep. I am running RefDB locally on a machine Dual PIII-700 with 512 MB
RAM using a postgres database on the same machine. So as it' s on the
same machine, it should not go through the network, shouldn't it ?
During the time I checked with top from time to time, and postgres
postmaster took between 40 and 90 percent.
Endnote swallows the data instantly (let' s say 5 secs)... :(


bd...@fa... said:
>> I've found the same.  Perhaps the two of us and Markus can figure out
>> a  solution? 

Yes, I'd love to.


Thanks for your support and input, Markus and Bruce.

Regards,
  Marc


-- 
 Dr. Marc Baaden - Laboratory of Molecular Biophysics, Oxford University
 mailto:ba...@sm...  - ICQ# 11466242 -  http://www.marc-baaden.de
 FAX/Voice +49 40333 968508  -  Tel: +44 1865 275380  or  +33 609 843217