refdb-devel — List for RefDB developers

Re: [Refdb-devel] Updated Emacs package, syntax highlighting, new name

[Michael S. <sm...@xm...>]

Bruce D'Arcus <bd...@fa...> writes:

> 
> Disregard request for customizing xml mode used for refdb output.   I 
> see my setup automatically uses nXML mode, which is all I wanted.

Well, I can still see some value to making parts of that configurable.
Yeah, for XML, it defaults to using nXML if you've got it installed, and
to PSGML if you don't; but others might want it the other way around.

  --Mike

Re: [Refdb-devel] Updated Emacs package, syntax highlighting, new name

[Michael S. <sm...@xm...>]

Bruce D'Arcus <bd...@fa...> writes:

> Looking really nice Mike!
> 
> I wonder: is there an appropriate mode to get better display of the 
> "screen" output?

Nothing I know of, unless Markus has ideas.

Actually, I wonder whether the default in Emacs should be something
other than the "screen" output -- maybe RIS? We'd be distributing the
refdb-mode.el package along with the ris.el package anyway, and to my
eyes at least, that package give a nice, readable, syntax-highlighted
view of the data.

But that is all said without having much insight in the use cases for
the various output types for the data. I guess I can see the advantages
of the 'screen' format when you're looking at the data in a shell, but
what's the value of viewing it that way in Emacs? Just the brevity?

> Also, could this later be configurable?
> 
> > - for HTML or DocBook SGML, sgml-mode (psgml)

Yes, I was just being lazy there. I can add config options to allow you
to specify which modes you prefer for XML and SGML and HTML output.
But let me know what modes you have in mind. These days, I only work
with XML and XHTML myself, and only use nXML, but I know other people
need to work with SGML and psgml, and maybe with other HTML modes.

  --Mike

Re: [Refdb-devel] refdb db in menu?

[Michael S. <sm...@xm...>]

hmm, I realize now that naming the base db "refdb" is arbitrary. I guess
a user could choose during install to call the base database "mybasedb"
or whatever, and then could create another database called "refdb", and
have references added to that.

Maybe I could make it configurable through a variable named
say, refdb-base-database. (In addition to the current
refdb-listdb-sql-regexp variable).

The code would take the list of databases that listdb returns, check it
against refdb-base-database, and remove any whatever name matches.  I
could make "refdb" the default, and document it by saying:

  Set this to whatever name you gave to the base RefDB database during
  installation (that is, using 'mysql -e "CREATE DATABASE refdb1"' or
  the PostgreSQL "createdb" command).

Or to make it more configurable, we could add refdb-listdb-ignore-list
variable, again with default value "refdb", but make it a list. Users
could put multiple names into it that way, and then wouldn't need to
mess around with trying to figure out how to construct a regex and
express it in whatever syntax their backend database supported.

And that way, we could also include the built-in database names for the
backends -- i.e. Postgres's "template0" and "template1" databases, and
whatever the names are of the built-in databases that MySQL sets up.

How does that sound?

Now that I write all that, though, I wonder whether that logic might not
be better handled on the RefDB side -- so that refdbd -C listdb wouldn't
return the name of the base RefDB database, whatever its name, or of the
built-in backend database names.

  --Mike

Michael Smith <sm...@xm...> writes:

> Markus Hoenicka <mar...@mh...> writes:
> 
> > Bruce D'Arcus writes:
> >  > 
> >  > Question:
> >  > 
> >  > Should the base db (now called "refdb") be included in the list of dbs 
> >  > in the menu?
> >  > 
> > 
> > I'm not sure how to handle this intelligently. In a shared environment
> > you'll always face the problem that your database engine hosts a
> > variety of databases that do not belong to RefDB. You should exclude
> > these as well. The only way that I can think of is to stick to a
> > name pattern for RefDB databases and run listdb with an appropriate
> > regexp. Maybe the menu code can use a regexp if one is set in .emacs?
> 
> Yes -- it actually already have the refdb-listdb-sql-regexp variable in
> there already to support just that. The default for it is "%".
> 
> But as far as the base db named "refdb" goes, users should never be
> directly adding any data to that via refdbc -C addref or whatever,
> right? If so, then I will have the code not add it to the menu.
> 
>   --Mike



-- 
Michael Smith
Openwave Systems Japan     +81 90 9148 0269 (mobile)
Nishishinjuku 6-22-1       +81  3 5909 6347 (office)
Tokyo, Japan 163-1117      +81  3 5909 6241 (fax)

Re: [Refdb-devel] refdb db in menu?

[Michael S. <sm...@xm...>]

Markus Hoenicka <mar...@mh...> writes:

> Bruce D'Arcus writes:
>  > 
>  > Question:
>  > 
>  > Should the base db (now called "refdb") be included in the list of dbs 
>  > in the menu?
>  > 
> 
> I'm not sure how to handle this intelligently. In a shared environment
> you'll always face the problem that your database engine hosts a
> variety of databases that do not belong to RefDB. You should exclude
> these as well. The only way that I can think of is to stick to a
> name pattern for RefDB databases and run listdb with an appropriate
> regexp. Maybe the menu code can use a regexp if one is set in .emacs?

Yes -- it actually already have the refdb-listdb-sql-regexp variable in
there already to support just that. The default for it is "%".

But as far as the base db named "refdb" goes, users should never be
directly adding any data to that via refdbc -C addref or whatever,
right? If so, then I will have the code not add it to the menu.

  --Mike

Re: [Refdb-devel] new-document and doc-processing menu behavior

[Bruce D'A. <bd...@fa...>]

On Dec 11, 2003, at 3:26 PM, Markus Hoenicka wrote:

> Michael Smith writes:
>> That makes sense. As I mentioned, I think also you might need to add
>> some logic for asking whether the existing doc uses short or full
>> notations, since there's not guarantee that the user has followed the
>> '*.short.*' convention that the current generated Makefiles expect for
>> short-notation documents.
>
> Ok, I'll rework the script along these lines.

Once support for the new biblioref element is added, the distinction 
between the two will evaporate; right?

Bruce

Re: [Refdb-devel] refdbc -C addref & stdin

[Markus H. <mar...@mh...>]

Bruce D'Arcus writes:
 > 
 > I'm not getting the first error anymore, but I am still getting the 
 > second (the "data read" error).
 > 

This essentially means that refdbc can't read from stdin. There's a
couple of things to check:

1) make sure that the commands run by refdb-menu do work on the
   command line. If not, it's probably a platform issue.

2) try adding the -f stdin switch to the appropriate code in
   refdb-menu.el

3) replace the refdbc call with a different program (maybe 'grep
   ".*"') to see whether the shell within Emacs correctly pipes the
   contents of the buffer to stdin of the called program.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-devel] refdbc -C addref & stdin

[Markus H. <mar...@mh...>]

Bruce D'Arcus writes:
 > 
 > On Dec 12, 2003, at 5:05 AM, Michael Smith wrote:
 > 
 > > Tried both with v0.9.3 and v0.9.2
 > 
 > I think Markus is going to say "you should be using the latest: 0.9.4 
 > pre-2"
 > 

Sort of. Using 0.9.4-pre1 over 0.9.3 is really recommended. 0.9.4-pre2
adds the still experimental notes stuff. It is at your discretion if
you want to fiddle with that.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-devel] refdbc -C addref & stdin

[Markus H. <mar...@mh...>]

Michael Smith writes:
 > Will it break behavior on other platforms if I just include the -f stdin
 > option by default in the set of args that Emacs passes to
 > refdbc -C addref ? That is, is -f stdin just ignored on platforms that
 > can already auto detect data on stdin?
 > 

It is safe to use this switch on all platforms. The switch isn't
actually ignored on the other platforms but it sets a variable to 1
which has the value 1 anyway if data are available on stdin.

 > Anyway, when I actually try it on Cygwin, it doesn't seem to work -
 > 
 >   $ refdbc -d foo -C addref -f stdin < /tmp/data/alltypes.ris 
 >   Could not set terminal attributes
 >   0 dataset(s) added, 0 skipped, 0 failed
 >   0 dataset(s) sent.
 > 

I'm afraid this is my fault. I've just checked that for reasons
entirely unclear to me addref does not yet support the -f stdin
switch. It will fail on Cygwin no matter how hard you'll try. The
appended patch fixes the problem. It is against a current CVS version
so expect some hunks to fail if you try to patch 0.9.3.

regards,
Markus

Re: [Refdb-devel] refdbc -C addref & stdin

[Bruce D'A. <bd...@fa...>]

On Dec 11, 2003, at 3:09 PM, Markus Hoenicka wrote:

>  you should set the password in your .refdbrc to
> circumvent this problem.

But what I'm saying is that setting it does not solve the problem for 
me.

Bruce

Re: [Refdb-devel] refdbc -C addref & stdin

[Bruce D'A. <bd...@fa...>]

On Dec 11, 2003, at 3:15 PM, Markus Hoenicka wrote:

>>> If I try to do it from the menu though (this with an risx file), I 
>>> get
>>> this:
>>>
>>> Could not set terminal attributes
>>
>> That's a warning coming from refdbc. I'm not sure what it indicates, 
>> but
>> I don't think it causes any problems.
>>
>
> It indicates that refdbc is asking for a password which is bound to
> fail. Please set the password in .refdbcrc.
>
>>> data read error. Stop.

I'm not getting the first error anymore, but I am still getting the 
second (the "data read" error).

Bruce

Re: [Refdb-devel] refdbc -C addref & stdin

[Bruce D'A. <bd...@fa...>]

On Dec 12, 2003, at 5:05 AM, Michael Smith wrote:

> Tried both with v0.9.3 and v0.9.2

I think Markus is going to say "you should be using the latest: 0.9.4 
pre-2"

Bruce

Re: [Refdb-devel] Updated Emacs package, syntax highlighting, new name

[Bruce D'A. <bd...@fa...>]

Disregard request for customizing xml mode used for refdb output.   I 
see my setup automatically uses nXML mode, which is all I wanted.

Bruce

Re: [Refdb-devel] refdbc -C addref & stdin

[Bruce D'A. <bd...@fa...>]

On Dec 12, 2003, at 5:05 AM, Michael Smith wrote:

> I also wish I had an OS X environment I could test with directly; seems
> like Bruce has found some problems that I haven't seen elsewhere.

There are two problems with emacs on os x:

1)  Apple released a new OS recently, which has significant 
under-the-hood changes.

2)  Because of this, old emacs binaries do not run on it, and I am 
forced to build from CVS, which is not always straightforward.  I was 
able to build without problems on this machine (my laptop), but have 
run into a few problems, including that I still cannot add a reference 
to refdb from the menu.  On my desktop, I couldn't even get emacs to 
build the other day, and the binary I downloaded that someone else 
built is flakey.

Markus, do you have any suggestions on figuring out the 
add-ref-from-menu thing?

Bruce

Re: [Refdb-devel] Updated Emacs package, syntax highlighting, new name

[Bruce D'A. <bd...@fa...>]

Looking really nice Mike!

I wonder: is there an appropriate mode to get better display of the 
"screen" output?

Also, could this later be configurable?

>  - for HTML or DocBook SGML, sgml-mode (psgml)

Bruce

[Refdb-devel] Updated Emacs package, syntax highlighting, new name

[Michael S. <sm...@xm...>]

Attached is an update of the Emacs package, renamed it "refdb-mode".
You'll need to change the require statement in your .emacs to:

  (require 'refdb-mode)

And any variables you have set in your .emacs will need to be changed
to, or example:

  (setq refdb-database "DATABASE")

That is, just drop the "menu-" part from the variable name.

Syntax highlighting
-------------------
  The main change in this version is that it now indirectly does syntax
  highlighting. It works just by trying to put the output buffer into an
  appropriate mode based on output type. For example:

    - for XML output types, it puts the output buffer into nxml-mode if
      you have it, sgml-mode (psgml) otherwise

    - for HTML or DocBook SGML, sgml-mode (psgml)

    - for BibTeX, bibtex-mode

    - for RIS, ris-mode

      http://cvs.sourceforge.net/viewcvs.py/*checkout*/refdb/refdb/site-lisp/ris.el

Window splitting
----------------
  I added a new variable, refdb-split-for-getref-output-flag. Default it
  non-nil, meaning Emacs will split the current frame in order to show
  the output buffer; setting it to nil means Emacs will show the output
  buffer at full frame. You can either set it from the customization
  buffer, or manually in your .emacs like this:

    (setq refdb-split-for-getref-output-flag nil)

  QUESTION: Should I make the default value in the code non-nil instead?

Separate buffer for RefDB messages
----------------------------------
  stderr from refdbc now goes to a separate *refdb-messages* buffer,
  instead of getting mixed with output. I've added a "Show RefDB Message
  Log" menu and refdb-show-messages command you can use to see that.

Commands
--------
  Command equivalents for all menu items are now available.

    refdb-addref-on-region
    refdb-getref-by-author
    refdb-getref-by-title
    refdb-getref-by-keyword
    refdb-getref-by-id
    refdb-getref-by-citekey
    refdb-getref-by-advanced-search
    refdb-select-output-type
    refdb-select-database
    refdb-show-messages

  I guess we should maybe discuss coming up with a set of key bindings.

I've also added a little more code to have Emacs emit more messages
about what it's doing; e.g., 

    (message "Adding references in selected region to %s database..." refdb-database)
    (message (format "Getting datasets for author %s ..." author))

  etc.

I made a lot of changes, so it might have broken some things. Please try
and let me know.

Next up, I'll probably be trying out the data caching and completion
stuff I mentioned, and then on from there to dealing with processing
docs/generating output.

  --Mike

Re: [Refdb-devel] refdbc -C addref & stdin

[Michael S. <sm...@xm...>]

Markus Hoenicka <mar...@mh...> writes:

> Michael Smith writes:
>  > um, has anybody actually had success at adding references to a RefDB
>  > database via the "Add Reference" menu item Emacs?

[...]

> Yes, me.
>  > None of those work for me. My refdbc doesn't seem to be able to read
>  > from standard input at all.
>  > 
> 
> Could it be you're working on Cygwin?

Yeah, sorry, I was when I wrote the message, and just hadn't bothered to
take the time to try it elsewhere. I should have done that before
sending the message.

> The data-waiting-on-stdin autodetection does not work on Cygwin,
> unfortunately, so if you want to do any of the above on Cygwin you
> have to use the -f stdin switch.

Ah, I see that now in the docs.

Will it break behavior on other platforms if I just include the -f stdin
option by default in the set of args that Emacs passes to
refdbc -C addref ? That is, is -f stdin just ignored on platforms that
can already auto detect data on stdin?

Anyway, when I actually try it on Cygwin, it doesn't seem to work -

  $ refdbc -d foo -C addref -f stdin < /tmp/data/alltypes.ris 
  Could not set terminal attributes
  0 dataset(s) added, 0 skipped, 0 failed
  0 dataset(s) sent.

Tried both with v0.9.3 and v0.9.2

It's not a big deal for my testing, because for now I've built a refdbc
binary from source that always reads from stdin no matter what, and set
up my Emacs so it calls that for running addref, rather than calling the
normal refdbc binary.

But for the long term, it would be good to make sure everything's
working on Cygwin as it should.

I also wish I had an OS X environment I could test with directly; seems
like Bruce has found some problems that I haven't seen elsewhere.

  --Mike

[Refdb-devel] Completion and caching

[Markus H. <mar...@mh...>]

Michael Smith writes:
 > Anyway, the more I consider this, I'm thinking maybe it makes a lot more
 > sense, regardless of how the alists are built, to do this:
 > 
 >   - run queries at RefDB startup to build in-memory alists of authors,
 >     titles, etc. and use those (instead of re-querying the db each time)
 > 
 >   - have a 'Refresh Data Cache' menu item ('refdb-refresh-data-cache'
 >     command) that you can manually run to update the local cache
 > 
 > How does that sound?
 > 

Sounds like a reasonable compromise to me.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

[Refdb-devel] Completion and caching

[Michael S. <sm...@xm...>]

Markus Hoenicka <mar...@mh...> writes:

> Hi Michael,
> 
> Michael Smith writes:
>  > Well, contents of those lists need not necessarily be in the menu itself
>  > in order to be made more easily accessible. It should be possible to
>  > come up with a mechanism for enabling users to do completion based on
>  > those. So, a user would select the Search > Author menu, then type the
>  > first letters of an author name -- "S" or whatever, then hit <tab> to
>  > complete against just the list of authors starting with S.
>  > 
> 
> Sounds good.
> 
> Another issue: how do you deal with the fact that the database might
> have changed between running two menu commands? Do you want to update
> the author (or keyword or periodical) list each time the menu is
> opened?

That's what I have in mind to do, but I haven't done it yet. Certainly,
it will have a performance difference. How much of a difference, I don't
know. Mostly it will depend on how quickly refdbc returns the data. But
also, once Emacs has the data, the code needs to take that and build an
alist out of it (the completing-read needs an alist, not a list).

I'm not sure if the time it takes Emacs to do that is significant or
not. The way I'm doing it now to make the Select Database menu is to
first build a list and then do this:

  (defun refdb-make-alist-from-list (list)
    "Make an alist from LIST by cons'ing elements with themselves."
    (mapcar
     (lambda (atom)
       (cons atom nil))
     list)
    )

With a big list, I think it might be significantly faster to instead do
it in one step. That is, just skip the step of building a list and
instead just build alist with what refdbdc returns.

Anyway, the more I consider this, I'm thinking maybe it makes a lot more
sense, regardless of how the alists are built, to do this:

  - run queries at RefDB startup to build in-memory alists of authors,
    titles, etc. and use those (instead of re-querying the db each time)

  - have a 'Refresh Data Cache' menu item ('refdb-refresh-data-cache'
    command) that you can manually run to update the local cache

How does that sound?

  --Mike

Re: [Refdb-devel] How to handle custom output types?

[Markus H. <mar...@mh...>]

Bruce D'Arcus writes:
 > Markus, how would custom stylesheets be used with RefDB?  In the Word  
 > example, all it does is use modified versions of the block and footnote  
 > xsl files, and then a special Word-tailored CSS file.  I really wish  
 > this was included in the default DocBook (x)html stylesheets as a  
 > parameter option, frankly.
 > 

Depends on whether you want to use the custom stylesheets as a default
or just once in a while. The latter isn't that easy, but using them
always amounts to pointing refdbxml to the modified versions instead
of to the stock DocBook stylesheets. You just need to edit the marked
section at the top of that script.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-devel] new-document and doc-processing menu behavior

[Markus H. <mar...@mh...>]

Michael Smith writes:
 > That makes sense. As I mentioned, I think also you might need to add
 > some logic for asking whether the existing doc uses short or full
 > notations, since there's not guarantee that the user has followed the
 > '*.short.*' convention that the current generated Makefiles expect for
 > short-notation documents.
 > 

Ok, I'll rework the script along these lines.

regards,
Markus


-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-devel] new-document and doc-processing menu behavior

[Markus H. <mar...@mh...>]

Michael Smith writes:
 > > If this is all it takes, then it's a no-brainer. I assume it doesn't
 > > hurt to use the --xinclude option even if the file does not contain
 > > Xincludes?
 > 
 > Right -- if you run it on a doc that doesn't actually have any Xincludes
 > in it, it just processes it the same way it would have if you hadn't
 > specified the --xinclude option at all.
 > 

Fine. I just added the switch to refdbxml.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

[Refdb-devel] XHTML output type?

[Markus H. <mar...@mh...>]

Michael Smith writes:
 > In an earlier message, Markus mentioned an 'xhtml' output type. But
 > unless there's some magic that I'm missing somewhere, there isn't
 > actually any XHTML output option, right?
 > 
 >   --Mike

getref -t xhtml should return xhtml output. If not, it's a bug.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-devel] refdbc -C addref & stdin

[Markus H. <mar...@mh...>]

Michael Smith writes:
 > Tonight was the first time I actually tried adding data to the db, and
 > environment I was trying it on is a Cygwin install. Haven't tried it on
 > my Linux install yet.
 > 
 > Anyway, got it working on Cygwin by hacking the refdbc source. So the
 > both the command line and Emacs interaction are working as expected for
 > me now.
 > 

The -f stdin switch should do the trick without source hacks.

 > > If I try to do it from the menu though (this with an risx file), I get 
 > > this:
 > > 
 > > Could not set terminal attributes
 > 
 > That's a warning coming from refdbc. I'm not sure what it indicates, but
 > I don't think it causes any problems.
 > 

It indicates that refdbc is asking for a password which is bound to
fail. Please set the password in .refdbcrc.

 > > data read error. Stop.
 > 
 > damn. Well. that one is also coming from refdbc too, and I think it's a
 > fatal error. If you're able to succesfully add data via the command line
 > but not from within Emacs, then either I'm not using the Emacs
 > shell-command-on-region function correctly, or there's something wrong
 > with the way your Emacs is interacting with the shell.
 > 

It works fine over here, but I haven't tested this on Cygwin yet. Do
you use Cygwin's Emacs or NT Emacs?

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

Re: [Refdb-devel] refdbc -C addref & stdin

[Markus H. <mar...@mh...>]

Bruce D'Arcus writes:
 > If I try to do it from the menu though (this with an risx file), I get 
 > this:
 > 
 > Could not set terminal attributes
 > data read error. Stop.
 > 

If refdbc asks for the password interactively it has to fiddle with
the terminal attributes in order to avoid echoing the password on the
screen. As you won't have a chance to enter a password anyway when
using the RefDB menu, you should set the password in your .refdbrc to
circumvent this problem. I've mentioned that way back in my original
post how to run refdbc from within Emacs.

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de

[Refdb-devel] refdb db in menu?

[Markus H. <mar...@mh...>]

Bruce D'Arcus writes:
 > 
 > Question:
 > 
 > Should the base db (now called "refdb") be included in the list of dbs 
 > in the menu?
 > 

I'm not sure how to handle this intelligently. In a shared environment
you'll always face the problem that your database engine hosts a
variety of databases that do not belong to RefDB. You should exclude
these as well. The only way that I can think of is to stick to a
name pattern for RefDB databases and run listdb with an appropriate
regexp. Maybe the menu code can use a regexp if one is set in .emacs?

regards,
Markus

-- 
Markus Hoenicka
mar...@ca...
(Spam-protected email: replace the quadrupeds with "mhoenicka")
http://www.mhoenicka.de