documancer-users — For users of Documancer

Re: [documancer] Re: [documancer-users] a few bugs?

[<arn...@we...>]

Hi,

first let me thank you very much for your response and the fixes (!).

On Wed, 24 Dec 2003, Vaclav Slavik wrote:

> arn...@we... wrote:
> >   The documancer files were put in
> >     ${HOME}/PYTHON_TST/lib/documancer
> >   where I would have expected them to be in
> >     ${HOME}/PYTHON_TST/lib/python2.3/site-packages/documancer
>
> Sorry, why? Documancer is *not* a Python package, it's an application,
> so why install it into the place where Python libraries are
> installed?

Well, my "expectation" came from other applications,
e.g. mayavi or pycrust, idle, helpviewer,...

> Also, you can currently run single Documancer installation
> using both 2.2 and 2.3 interpreter, but you'd have to have two
> installations

That's a good point, indeed.

> >   (Actually, a setup.py file might be nice).
>
> While setup.py may be useful for installing multi-platform Python
> packages, I don't find it terribly useful for installation of
> applications. In fact I consider autoconf's checks much more useful
> -- is there some way to seemlessly integrate autoconf and setup.py?
> Also, note that Documancer is mixed-language code (C and Python).

I don't know distutils well enough, but I heard about
several people/projects being pretty unhappy with it in many respects.

Anyway, I did not have any problems in installing documancer,
so maybe it's better to leave this aside, there are more
interesting and more important topics to be delt with ;-)


> > - When clicking on a link to a ps or pdf document
> >   there is always a kind of download window,
> >   titled wxmozilla. It contains a list of all the
> >   previous .ps/.pdf.
> >   Is there a way to (optionally)
> >   prevent this window from showing up at all?
>
> I don't know. But I'm [almost] sure it can't be done on wxMozilla
> level.

Hmm, I really don't like it so I will have a closer look at
that (not questioning your expertise here, but sometimes
naive user questions, ignorant of implementation specific
details, may encourage developers (wxMozilla) for changes ;-)

So far I tried to set in ~/.documancer/mozilla-profile/prefs.js
  user_pref("browser.download.progressDnldDialog.keepAlive", false);
which is "accepted" (according to about:config)
but does not give the desired effect (maybe it is
the wrong option ;-).

> >   (even though it might be sometimes nice as it allows to inspect
> >    the settings for wxmozilla with about:config)
>
> This inspired me to add "Configure Mozilla" menu item that opens
> about:config, but it doesn't work well (read-only, doesn't workd &
> even crashes when you try to edit the prefs).

That looks nice - also checked it here: same happens ....
(BTW: maybe it is problematic to link the mozilla configuration
from ~/.mozilla to the one from documancer as suggested in the FAQ
as changes on the documancer side reflect back to the "normal"
mozilla (which might be even a different version) ...)


[...]
> >   Therefore it would be nice to have the search indices
> >   for these documents just generated once (by "root")
> >   so that the users to not have to create and store
> >   their own copy of this.
>
> Yes... It opens some problems, though:
> * How do you edit global vs. root-user-only books?

sorry, I don't understand the question -  what do
you mean by "global" books ?

> * Provided Documancer can update index if it detects its out of date
> (it will support it, eventually),

Great - looking forward to that ...

> how can it update shared books if
> it is not run as root?

Hmm, maybe this is asking too much: the program could just say: "sorry,
I cannot update the index, please ask root to do it...".
Optionally it could offer to create a local index
(which gets removed if there is an updated global one ??)
Generally I would think that documentation on the root
level will not change that often ...

Anyway, you are absolutely right, this requires a bit of thinking ...


> > - In addition to html files there might be also pdf files.
> >   Would it be possible to include these into the search
> >   when they are referenced from a html file?
>
> Not easily -- I can't index PDF files currently, I can only display
> them if Acrobat Reader is installed, I can't highlight search results
> in them, have bookmarks, i.e. they cannot be first class citizens in
> Documancer yet. But I'd like to be able to fully use them at some
> point (by means of xpdf or something similar).

Personally I would say that indexing PDF files would be
a big step forward. Searching inside the PDF file would
then be just done via acroread's search.
So it would not be a first class or third class ;-) citizen
but maybe second class. Fine enough as more than a start IMHO ;-).

By now I had a look at swish-e and actually
one can tell it to index pdf files
(or anything else) by providing the appropriate filters.

I tried the prog variant (i.e. example9.config from
the swish-e examples), however I ended up with
"Warning: XML parse error in file".
In contrast  example8.config, which uses a filter,
_pdf2html.pl and the xpdf package , works fine!
((The call is just
swish-e  -c example8.config -v 3  -f index.swish-e
/directory_with_html_and_pdf/
))
In contrast the call
swish-e -v 3  -f index.swish-e -i    /directory_with_html_and_pdf/
will also index the pdf files, but "raw", i.e. without
conversion to html or xml, producing useless index entries.
If you want I can look further into this
so that one could add this to the configuration file
written indexer.py.

> > - Thinking of having a documancer session open all the time
> >   it would be nice if there is a way to
> >   direct queries from an interactive (I)python session
> >   (or pycrust, or emacs, or ...)
> >   to documancer.
>
> Also planned...

Alright, I also looked into this a bit - I have set up a small
example which communicates via sockets
To incorporate this into documancer the approach could be the following
 - app.py: DocumancerApp(wxApp):
    def OnInit(self):
        [....]
        docuServer.runDocuServer(inBackground=1,DoSearch)

   # - here DoSearch is defined in MainFrame.py
   # Question: should one use the uiCallback for making
   #           this available here ?

 - MainFrame.py:

    def DoSearch(self,query):
       [... old OnSearch ...]

    def OnSearch(self, event):
        DoSearch(self.search_text.GetValue())

Now any external programm which wants to send a search string
needs to know the address of the "docu server"
(determined analogously to getServerPort (from utils.py)).
Should one store that number in a file in the ~/.documancer
directory?
What do you think, does the above sound like a reasonable
approach ? If you want I can try to piece the things
together...
Hmm, there is one missing bit in the above approach:
in some way we need to specify the book in which the search should be
done ...
(so maybe the external program needs to parse ~/.documancer/config.ini,
and pre-pend this to the search string ?).

> But the TODO with my plans is way too big, so I could
> definitely use some help, as usual ;)

Honestly, I would love to help out more, but presently
I am really swamped with other (scientific) stuff
which only leaves me little time to look into python things
(and I have couple of small python things which eat up that "spare" time).
So at this point I better don't make any promises
and just restrict myself to some kind of "testing".

Best,

Arnd

Re: [documancer] Re: [documancer-users] a few bugs?

[Vaclav S. <vac...@ma...>]

Hi,

arn...@we... wrote:
>   The documancer files were put in
>     ${HOME}/PYTHON_TST/lib/documancer
>   where I would have expected them to be in
>     ${HOME}/PYTHON_TST/lib/python2.3/site-packages/documancer

Sorry, why? Documancer is *not* a Python package, it's an application,=20
so why install it into the place where Python libraries are=20
installed? Also, you can currently run single Documancer installation=20
using both 2.2 and 2.3 interpreter, but you'd have to have two=20
installations=20

>   (Actually, a setup.py file might be nice).

While setup.py may be useful for installing multi-platform Python=20
packages, I don't find it terribly useful for installation of=20
applications. In fact I consider autoconf's checks much more useful=20
=2D- is there some way to seemlessly integrate autoconf and setup.py?=20
Also, note that Documancer is mixed-language code (C and Python).

> - Books manager:
>   starting afresh (i.e. no books added yet)
>   I first entered a Book title, selected Html
>   but did not get the URL field.
>   Only then I realized that I have to press "New" first
>   and only then one can add the book.
>   Ok, this is user stupidity ;-), but maybe one should either
>     a) disable the entry fields first
>     b) or allow for adding a book
>       by just typing the information
>       and accept this by pressing
>       an "Add" button.
>
>   Also by pressing "New" a "book's name"
>   entry box opens even though there
>   is entry field for this
>   (Named "Book title" vs. "book's name").
>   To summarize: I think it would be more intuitive
>   to go for variant b).

I agree completely, but it will take a little more time to do this.

> - after adding the first book and returning from the Books manager
>   the index page of this document is not shown.
>   (One has to press "Return to homepage" explicitly).

Thanks, will fix.

> - when doing a search the first hit could be shown
>   automatically (maybe as option)

Done now. I'll make it an option if somebody complains about it;)

>   (also the width of "Score" and "Document" is not
>    well adapted and changing the width of the Search/Bookmarks
>    part the widths of "Score"/"Documents" is not changed
> accordingly)=20

What do you mean? Score column has fixed width here and the Document=20
one has width of the widest page title, is there something wrong with=20
that or does it behave differently for you?

> - clicking on a result after a search opens
>   the page of that document, but does not jump to
>   the position of the first hit in that document

=46ixed as well.

> - after a search: buttons (and key shortcuts)
>   to move to the next/previous hit in the document

Will do shortly, it's been planned, but I had to fix some problems=20
with the highlighter first (now done).

> - searches across several books would be useful as well
>   (e.g. with a tick-list to specify which
>    books should be included in the search)

Yes. I want it to be even better: I want it to let you define=20
collections of books and bind them to specific directories or source=20
files on your disk, so that you can invoke Documancer from text=20
editor and it will know which books are relevant to the file you're=20
editing.

> - it would be nice if updating/generating an index is
>   done in a separate thread (+ a busy indicator maybe ...).
>   (Then one could at least continue browsing in the meantime ...)

Planned (see the TODO file in CVS). I also want it to automatically=20
detect that the content changed and reindex it.

> - When clicking on a link to a ps or pdf document
>   there is always a kind of download window,
>   titled wxmozilla. It contains a list of all the
>   previous .ps/.pdf.
>   Is there a way to (optionally)
>   prevent this window from showing up at all?

I don't know. But I'm [almost] sure it can't be done on wxMozilla=20
level.

>   (even though it might be sometimes nice as it allows to inspect
>    the settings for wxmozilla with about:config)

This inspired me to add "Configure Mozilla" menu item that opens=20
about:config, but it doesn't work well (read-only, doesn't workd &=20
even crashes when you try to edit the prefs).

> - "About": you could include a link to the web-page
>   (so interested users can get straight to all further information,
>    e.g. mailing list, updates etc. ...)

It's been there, but well-hidden, now fixed.

> Wish-List points:
> - Support for a tree view of Table of Contents for the books
>   a la wxHtmlHelper, like e.g. for the wxpython doku.

Definitely planned (and not only for .hhp, but for gtk-doc and others=20
as well), but I didn't have time to do it yet.

>   ((Actually, I think that will be the last open point
>   to make me switch from helpviewer.py to documancer ;-))

Nice to hear :)

> Some meta stuff (not thought to an end ;-):
> - I am thinking of using documancer as _the_ documentation
>   center for our course on computational physics next summer term.
>   So there will be several users accessing the same base of
> documents (python documentation, dive-into-python tutorial, Numeric
> manual, scipy documentation, wxPython manual, ...).
>   Therefore it would be nice to have the search indices
>   for these documents just generated once (by "root")
>   so that the users to not have to create and store
>   their own copy of this.

Yes... It opens some problems, though:
* How do you edit global vs. root-user-only books?
* Provided Documancer can update index if it detects its out of date=20
(it will support it, eventually), how can it update shared books if=20
it is not run as root?

> - In addition to html files there might be also pdf files.
>   Would it be possible to include these into the search
>   when they are referenced from a html file?

Not easily -- I can't index PDF files currently, I can only display=20
them if Acrobat Reader is installed, I can't highlight search results=20
in them, have bookmarks, i.e. they cannot be first class citizens in=20
Documancer yet. But I'd like to be able to fully use them at some=20
point (by means of xpdf or something similar).

> - Thinking of having a documancer session open all the time
>   it would be nice if there is a way to
>   direct queries from an interactive (I)python session
>   (or pycrust, or emacs, or ...)
>   to documancer.

Also planned... But the TODO with my plans is way too big, so I could=20
definitely use some help, as usual ;)

Regards,
Vaclav

=2D-=20
PGP key: 0x465264C9, available from http://pgp.mit.edu/

Re: [documancer] Re: [documancer-users] a few bugs?

[<arn...@we...>]

Hi Vaclav,

On Thu, 11 Dec 2003, Vaclav Slavik wrote:

> Hi,
>
> (sorry for late reply, I was away during the weekend and was a bit
> busy ever since)

same with me, just to much to be done before christmas ;-),
but here we go ...

> > c) For pydoc: site-packages/wxPython/wx:
> >     either: crashes
> >     or: mouse pointer with watch - application hangs
> >
> >     However, other texts work, so this might be caused by wx ?
>
> Rather by the way pydoc works -- it imports the modules and this may
> cause problems if the module does some initialization immediately
> when imported. The only safe fix seems to be to abandon pydoc and use
> epydoc or HappyDoc to generate documentation from Python sources.
> That or running pydoc in separate process.

epydoc or HappyDoc might be worth a try

Thanks for the fixes!

And now to a couple of further points:

- Place of the installed documancer:
  When installing from CVS I did
    export PHOME=${HOME}/PYTHON_TST/
    ./autogen.sh
    ./configure --prefix=${PHOME}
    make
    make install

  The documancer files were put in
    ${HOME}/PYTHON_TST/lib/documancer
  where I would have expected them to be in
    ${HOME}/PYTHON_TST/lib/python2.3/site-packages/documancer

  (Actually, a setup.py file might be nice).

- Books manager:
  starting afresh (i.e. no books added yet)
  I first entered a Book title, selected Html
  but did not get the URL field.
  Only then I realized that I have to press "New" first
  and only then one can add the book.
  Ok, this is user stupidity ;-), but maybe one should either
    a) disable the entry fields first
    b) or allow for adding a book
      by just typing the information
      and accept this by pressing
      an "Add" button.

  Also by pressing "New" a "book's name"
  entry box opens even though there
  is entry field for this
  (Named "Book title" vs. "book's name").
  To summarize: I think it would be more intuitive
  to go for variant b).

- after adding the first book and returning from the Books manager
  the index page of this document is not shown.
  (One has to press "Return to homepage" explicitly).

- when doing a search the first hit could be shown
  automatically (maybe as option)
  (also the width of "Score" and "Document" is not
   well adapted and changing the width of the Search/Bookmarks
   part the widths of "Score"/"Documents" is not changed accordingly)
- clicking on a result after a search opens
  the page of that document, but does not jump to
  the position of the first hit in that document
- after a search: buttons (and key shortcuts)
  to move to the next/previous hit in the document
- searches across several books would be useful as well
  (e.g. with a tick-list to specify which
   books should be included in the search)
- it would be nice if updating/generating an index is
  done in a separate thread (+ a busy indicator maybe ...).
  (Then one could at least continue browsing in the meantime ...)
- When clicking on a link to a ps or pdf document
  there is always a kind of download window,
  titled wxmozilla. It contains a list of all the
  previous .ps/.pdf.
  Is there a way to (optionally)
  prevent this window from showing up at all?
  (even though it might be sometimes nice as it allows to inspect
   the settings for wxmozilla with about:config)
- "About": you could include a link to the web-page
  (so interested users can get straight to all further information,
   e.g. mailing list, updates etc. ...)


Wish-List points:
- Support for a tree view of Table of Contents for the books
  a la wxHtmlHelper, like e.g. for the wxpython doku.

  I.e. this would mean support for the .hhp files
  (from the.hhc/.hhk/.hhp combo)

  ((Actually, I think that will be the last open point
  to make me switch from helpviewer.py to documancer ;-))

Some meta stuff (not thought to an end ;-):
- I am thinking of using documancer as _the_ documentation
  center for our course on computational physics next summer term.
  So there will be several users accessing the same base of documents
  (python documentation, dive-into-python tutorial, Numeric manual,
  scipy documentation, wxPython manual, ...).
  Therefore it would be nice to have the search indices
  for these documents just generated once (by "root")
  so that the users to not have to create and store
  their own copy of this.

- In addition to html files there might be also pdf files.
  Would it be possible to include these into the search
  when they are referenced from a html file?

  This might open a can of worms - I.e.,
  if one allows for external files how could documancer
  know that a rebuilt of the search index is
  necessary if any of the files has changed?

- Thinking of having a documancer session open all the time
  it would be nice if there is a way to
  direct queries from an interactive (I)python session
  (or pycrust, or emacs, or ...)
  to documancer.

OK, enough for this time,

Arnd

[documancer] a few bugs?

[Vaclav S. <vac...@ma...>]

Hi,

Vaclav Slavik wrote:
> Yes, sure -- all I need to do is to somehow (I didn't try to find
> out how yet, so if you know, tell me!) detect unused port(s) and
> use it instead of fixed one.

OK, this now works in CVS, thanks again for pointing it out.

Vaclav

=2D-=20
PGP key: http://pgp.mit.edu:11371/pks/lookup?op=3Dget&search=3D0x465264C9

[documancer] Re: [documancer-users] a few bugs?

[Vaclav S. <vac...@ma...>]

Hi,

(sorry for late reply, I was away during the weekend and was a bit=20
busy ever since)

Arnd Baecker wrote:
> When doing a few tests with documancer I observed
> a couple of points (I don't know if they have been mentioned
> before, but on sourceforge
> there is no archive to the mailing list yet).

That's because you're the first one to post to it ;)

> a) It would be nice to have the "Adding Books" section and
>    the "Keyboard shortcuts" of the README
>    accessible from the "Help" menu of the documancer

You're right -- I didn't do it yet because there are so many things to=20
do in Documancer...

> b) Adding a book:
>    I did: File/Manage Books/New Book,
>     HTML: /home/python/PYTHON_TST/docs/Python-Docs-2.3.2/index.html
>    and enabled fulltext search.
>
>     After this the corresponding Book "PythonHtml"
>     is the active book is only listed when one clicks on
>     the "books selection"
>
>     Typing "print" in the search field gives:
>
>   File "/home/python/PYTHON_TST/lib/documancer/gui/MainFrame.py",
> line 313, in OnSearch
>     for title, url, score in indexer.search(self.currentBook,
> query): File "/home/python/PYTHON_TST/lib/documancer/indexer.py",
> line 121, in search
>     __ensureBookIndexValidity(b)
>   File "/home/python/PYTHON_TST/lib/documancer/indexer.py", line
> 108, in __ensureBookIndexValidity
>     if b.getAttr(book.ATTR_DIRTY) =3D=3D '1':
> AttributeError: 'NoneType' object has no attribute 'getAttr'

If the book wasn't selected, right? Thanks, I fixed it CVS -- it=20
should no longer be possible to end up w/o some book selected unless=20
you don't have any books at all.

>     One minor point: changing "full-text search" to disabled in the
>     Bookmanager gives when no book is selected (the first time)
>
> Traceback (most recent call last):
>   File
> "/home/python/PYTHON_TST/lib/documancer/gui/BooksManagerDialog.py",
> line 161, in OnSettingsChange
>     self.UpdateBook()
>   File
> "/home/python/PYTHON_TST/lib/documancer/gui/BooksManagerDialog.py",
> line 221, in UpdateBook
>     self.SetConfigPanel(book.provider)
>   File
> "/home/python/PYTHON_TST/lib/documancer/gui/BooksManagerDialog.py",
> line 143, in SetConfigPanel
>     panel =3D providers.providers[provider].\
> KeyError: ''

Thanks, fixed as well.

> c) For pydoc: site-packages/wxPython/wx:
>     either: crashes
>     or: mouse pointer with watch - application hangs
>
>     However, other texts work, so this might be caused by wx ?

Rather by the way pydoc works -- it imports the modules and this may=20
cause problems if the module does some initialization immediately=20
when imported. The only safe fix seems to be to abandon pydoc and use=20
epydoc or HappyDoc to generate documentation from Python sources.=20
That or running pydoc in separate process.

> d) Searching for something with pydoc: it seems that
>    generating the index  does not come to an end.

It's possible that it takes *really* long to index everything. Or that=20
it hands when it tries to fetch wxPython files, see c).

> e) Presently it seems that it is not possible that
>    two different users run documancer on the
>    same machine ("Address already in use").
>    In the longer run I would like to run documancer on a cluster
>    with several users logged in. Is there any possibility for this
>    in principle?

Yes, sure -- all I need to do is to somehow (I didn't try to find out=20
how yet, so if you know, tell me!) detect unused port(s) and use it=20
instead of fixed one.

> I also have a couple of wish-list points, however they need
> a bit more thinking ...

Looking forward to them ;)

Thanks,
Vaclav

=2D-=20
PGP key: http://pgp.mit.edu:11371/pks/lookup?op=3Dget&search=3D0x465264C9

[documancer-users] a few bugs?

[Arnd B. <arn...@we...>]

Hi,

first: many thanks - this documancer is really great!!!
(A big improvement over the wxHtmlHelpController)

When doing a few tests with documancer I observed
a couple of points (I don't know if they have been mentioned
before, but on sourceforge
there is no archive to the mailing list yet).

a) It would be nice to have the "Adding Books" section and
   the "Keyboard shortcuts" of the README
   accessible from the "Help" menu of the documancer

   (At first I was lost what to do after starting documancer ;-)

b) Adding a book:
   I did: File/Manage Books/New Book,
    HTML: /home/python/PYTHON_TST/docs/Python-Docs-2.3.2/index.html
   and enabled fulltext search.

    After this the corresponding Book "PythonHtml"
    is the active book is only listed when one clicks on
    the "books selection"

    Typing "print" in the search field gives:

  File "/home/python/PYTHON_TST/lib/documancer/gui/MainFrame.py", line
313, in OnSearch
    for title, url, score in indexer.search(self.currentBook, query):
  File "/home/python/PYTHON_TST/lib/documancer/indexer.py", line 121, in
search
    __ensureBookIndexValidity(b)
  File "/home/python/PYTHON_TST/lib/documancer/indexer.py", line 108, in
__ensureBookIndexValidity
    if b.getAttr(book.ATTR_DIRTY) == '1':
AttributeError: 'NoneType' object has no attribute 'getAttr'

    and the watch appears (forever ?) but the menus are still accessible.
    Exiting and restarting documancer it does work.
    Also the starting page of the Book PythonHtml  is visible then.

    Searching also works then
    (it creates the index which takes a short while and
    then searching anything else is amazingly fast !!!)

    One minor point: changing "full-text search" to disabled in the
    Bookmanager gives when no book is selected (the first time)

Traceback (most recent call last):
  File "/home/python/PYTHON_TST/lib/documancer/gui/BooksManagerDialog.py",
line 161, in OnSettingsChange
    self.UpdateBook()
  File "/home/python/PYTHON_TST/lib/documancer/gui/BooksManagerDialog.py",
line 221, in UpdateBook
    self.SetConfigPanel(book.provider)
  File "/home/python/PYTHON_TST/lib/documancer/gui/BooksManagerDialog.py",
line 143, in SetConfigPanel
    panel = providers.providers[provider].\
KeyError: ''


c) For pydoc: site-packages/wxPython/wx:
    either: crashes
    or: mouse pointer with watch - application hangs

    However, other texts work, so this might be caused by wx ?

d) Searching for something with pydoc: it seems that
   generating the index  does not come to an end.

e) Presently it seems that it is not possible that
   two different users run documancer on the
   same machine ("Address already in use").
   In the longer run I would like to run documancer on a cluster
   with several users logged in. Is there any possibility for this
   in principle?


I also have a couple of wish-list points, however they need
a bit more thinking ...

Arnd


System remarks:
---------------

- debian linux
- python 2.3.2
- wxPython/wxWindows from CVS
- mozilla 1.5.1
- wxmozilla from CVS
- documancer-0.2.2

Compilation of all packages with: gcc 3.3.2