#707 Html help/doc via GUI terminal

[Dan Sebald]

There was mention in another patch thread

https://sourceforge.net/p/gnuplot/patches/704/

about integrating gnuplot's help as a HTML resource into the WXT terminal. I've tried this with some positive preliminary results (but I'm not yet familiar with the docs material) so this is a new thread to follow through on developing the notion. I also notice there are all kinds of figures that are processed (and failed), but I think those might be the demo examples.

What I've done is run "make html" in the source tree and the result is a ./docs/htmldocs/gnuplot.html along with many supporting files in the directory. Upon pressing the "about" button (with the question mark graphic) I've created a simple HTML window and loaded this file using the full directory path. Attached is a low-res screen shot of the result. The HTML paths all seem to work fine.

I'm not sure if this is the material that was being asked about. Does this look like the right idea? Does the window need to be embellished in some way? What about the "about" button? Does that need to be enhanced so there is an "about" and a "help"? In the wxWidgets documentation there is same discussion about an even more feature-rich version of the "html help" window which supports Windows help format. I sort of recall this Windows help from once having done such a thing in the 1990s, but people will have explain a bit more what they have in mind. (Seems to require bundling all files into a zip file.)

The other part of this is that in order to get help available at the terminal, we need to put the HTML files somewhere on the system and have some mechanism within gnuplot for accessing that path. "make install-html" doesn't appear to do anything. Ethan can probably give some suggestions on how to install the help files and access gnuplot.html via C code.

Also, I believe Qt has similar mechanisms for HTML code and I will try to do parallel code in both WXT and Qt terminals at the same time.

[Karl Ratzsch]

The windows makefile creates everything necessary (hhc/k/p files containing the html code, images) in docs/windows/, where the MS Help Compiler packs it into wgnuplot.chm. Packing that directory into a zip instead should do the trick, if i understood this here

http://docs.wxwidgets.org/trunk/overview_html.html#overview_html_helpformats

correctly. I think the relevant part of the mingw makefile should also run on linux (except for the last step), if docs/windows/doc2html.c compiles on linux.

[Bastian Märkisch]

No, it is actually very simple, since a make target already exists. To get a complete wxWidgets htb help file which includes graphics, just run make htb in the docs directory. This will create gnuplot.htb. Depending on your compiler you might need to change the variable named basename in docs/windows/doc2html.c to something else, though.

[Dan Sebald]

I ran across some bugs:

1) "basename" collides with something from the C++ strings class, I believe. In doc2html.c I changed "basename" to "wbasename".

2) Got the following error:

gcc -g -O2 -L/usr/lib64 -o wxhelp/doc2html wxhelp/doc2html.o termdoc.o xref.o ../src/version.o
./wxhelp/doc2html ../../gnuplot/docs/gnuplot.doc wxhelp/
GNUPLOT_LIB=../../gnuplot/docs/../demo ../src/gnuplot -e "winhelp=1;" ../../gnuplot/docs/plotstyles.gnu

set output out . 'figure_lines' . ext
^
cannot open file; output not changed
"../../gnuplot/docs/plotstyles.gnu", line 41: util.c: No such file or directory

make: *** [gnuplot.htb] Error 1

The expression for the file evaluates to "./windows/figure_lines.png". However, there is no subdirectory "./windows" so the "set output" step fails. All I did to make this work is manually create the subdirectory "windows". After that, the make process for htb progressed as designed.

3) The gnuplot.htb file is missing the "gnuplot.hhp" header file. I created such a file with the contents:

Contents file=wgnuplot.hhc
Index file=wgnuplot.hhk
Title=Book of Knowledge (ommm)
Default top=wgnuplot.html

and placed that inside the zip archive gnuplot.htb. However, this did not work correctly. I then unzip the contents of the gnuplot.htb file into its own subdirectory. That worked, and I've attached a low resolution image of the result. It took a bit of extra work to get things to work with the gnuplot.htb file, but it was basically that upon initialization of the wxWidgets the file system has to be set up to handle archival files.

OK, so I have the Windows book help working, but it won't be for several days that I have a chance to look at this again and start to make a workable patch file. In the mean time, if one of you would like to update the "make htb" process to fix the above three bugs, that would be helpful.

I would add one more thing that could be done, which is

4) Add the "make htb" to the root level Makefile process. However, I think Ethan's help will be needed to have the Windows book ZIP file installed somewhere as part of the "make install" step.

[Bastian Märkisch]

1) Is fixed in CVS for 5.0 and 5.1
2), 3) I suppose your CVS checkout is incomplete. There really is a docs/windows directory which also contains the gnuplot.hhp file. It is also required for the build process on Windows.

Btw. running make htb in the docs directory works fine here on CentOS 5.

[Dan Sebald]

Just some notes on what I've found for Qt.

There are several options.

1) In Qt there is a fair amount of support for working with HTML code. I think a file can be read in and interpreted as HTML and displayed. This would be similar to the first example of the interactive gnuplot manual I gave.

2) Qt has its own feature that is similar to MS HTML manual. It is called Qt Assistant as a stand alone application. I believe it can be accessed through C++ code. Try "assistant-qt" in a shell window. There is a utility for building a help project

http://qt-project.org/doc/qt-4.8/qthelpproject.html

with a syntax as follows:

qhelpgenerator doc.qhp -o doc.qch

We'd need a "make qch" equivalent to "make htb", but that's an amount of work I don't have time for at the moment.

3) I think there is a sort of web-browser kit in Qt, so if the gnuplot manual is online somewhere, that could be displayed in a window.

4) There is a Qt-based utility called kchmviewer

http://www.ulduzsoft.com/linux/kchmviewer/

which accepts a chm (compressed helped manual) file. CHM must be an open-source similarity to MS HTML help manual. I installed kchmviewer on my system and it looks very similar to, no surprise, Qt Assistant or MS HTML help, or WXT htb viewer.

My preference would be number 2, utilizing Qt's assistant or help framework. Just not enough time right now to build the QHP file from the HTML code. So lets concentrate on WXT, because that one is so close, and save Qt for a separate patch thread.

[Dan Sebald]

Attached is an evaluation-ready patch for using the gnuplot doc files in both the Help Book feature of the wxWidgets framework used by WXT terminal and the Qt-Assistant feature of the Qt framework used by Qt terminal.

This should be straightforward to compile and test. To get a feel for how things are programmed, I suggest the following:

• Compile and install the patched gnuplot tree as usual, i.e., "make install".
• Run gnuplot and try a sample plot in both the Qt and WXT terminal.
• With the sample terminal visible click on the "Manual" icon in
  the toolbar menu. You will get a message about missing doc files.
• Compile the HTML version of the doc files, i.e., "make html"
• Install the HTML version of the doc files, i.e., "make install-html"
• Again try sample plots in both the WXT and Qt terminal, click on
  the "Manual" icon.
• Install the Help Book code: "make htb", "make install-htb"
• Install the Qt-Assistant help collection: "make qhc", "make install-qhc"
• Again try sample plots in both the WXT and Qt terminal, click on the
  "Manual" icon.

I think if one looks at the patch file in a good highlighting editor, it should be evident what all the changes do. There is no tricky programming of any sort--I just followed the paradigm of the framework. There was some tedious programming to create the correct HTB and QHC files, but all it amounts to is a few lines here and there in the doc2html.c program.

I changed the organization of some subdirectories in the /docs directory because they made more sense to me. For example, the HTB files, from our perspective, are not restricted to Windows. I renamed the /windows directory to /helpbook because that's now where the HTB and QHC files are generated (before being moved to new directories /wx and /qt, respectively). The doc files for Qt-Assistant end up in /qt, the doc files for HelpBook end up in /wx. (As before, the HTML files end up in /html subdirectory.)

I made the installed documentation subdirectories reflect the above organization in /docs subdirectory. So, upon installing, for example, it might look like:

[@build5]# ls /usr/local/share/gnuplot/5.1/help/
html qt wx

I also changed the "wgnuplot.*" naming convention because, again, the "w" implied Windows and that really isn't generic enough. I tossed the "w" on a few files and renamed "wgnuplot.html" to "mainpage.html" because that is really what its purpose is inside the HelpBook/help-collection. A second motivation for that rename is that the HTML documentation (under the "html" subdirectory) has a "gnuplot.html" for which I wanted to avoid conflict.

In summary, once ready for CVS checkin, the following file maneuvers should be done:

Remove: docs/windows/doc2html.c
Remove: docs/windows/wgnuplot.hhk
Remove: docs/windows/wgnuplot.hhp
Remove: docs/windows/wgnuplot.stp

Add: docs/helpbook/doc2html.c
Add: docs/wx/gnuplot.hhk
Add: docs/wx/gnuplot.hhp
Add: docs/wx/gnuplot.stp

Add: docs/qt/gnuplot.qhcp

There are a couple bugs, I believe, that I fixed along the way. One had to do with compilation

1. Force the ./docs/helpbook (./docs/windows) subdirectory to be created in docs/Makefile.am if the directory doesn't exist. Here is the mod in the pre-patch code in case someone wants to make the "-p" addition in the 4.8/5.0 code:

   gnuplot.htb: wxhelp/wgnuplot.html
   GNUPLOT_LIB=$(srcdir)/../demo ../src/gnuplot -e >"winhelp=1;" $(srcdir)/plotstyles.gnu
   mkdir -p windows

2. I think that a set of tags was missing from the gnuplot.hhc at the very end of the list. The ramification in the help book compressed file was that all the sections appeared to be its own help book, so the contents window showed a half dozen different books. After providing those closing tags, the sections then appeared all under the one help book "gnuplot". Basically, I fixed this by calling the routine one more time with a phantom entry that prints no new section tag, i.e.:

   /* process each line of the file */
   while (get_line(line, sizeof(line), a)) {
   process_line(line, b, c, d, e);
   }
   /* One more phantom line to close out any levels.
   * A level of zero is treated similar to 1, but
   * no new field is created. */
   process_line("0", b, c, d, e);

The above is probably the trickiest code in the patch on first glance.

I will attach a couple PNG snapshots of the WX HelpBook and the Qt-Assistant on the same page, so you know what to compare against. Note that the 3-D plot looks kind of strange, with the left-most vertical axis sort of fragmented. I don't know if that is a bug or was as intended.

It's possible to do some customization still on the Qt-Assistant files. See:

http://doc.qt.io/qt-5/assistant-custom-help-viewer.html

For example, we could add a note for the popup "About" menu. We can change the icon of the Qt Assistant window to be one that is gnuplot specific, but I really don't want to use an icon that is the same as the Qt terminal window. I'd prefer something more distinct for windows that have a distinct behavior. The "Qt" icon is fine for now.

Here's a question: Why is the makefile.dst file so old? It's date is January 22, 2000. If this file is being used in some way, I would guess it would need to have been updated at some point since then.

[Dan Sebald]

WXT terminal snapshot...

[Dan Sebald]

QT terminal snapshot...

[Karl Ratzsch]

The changes to docs/Makefile.am don't apply cleanly any more. Could we have a new patch? I don't think I can make it myself in any reasonable amount of time.

I find this really cool. One thing i sorely miss in linux is wgnuplots online help. The pdf isn't nearly as comfortable, not speaking of the console help.

[Dan Sebald]

Updated patch attached. Quite a few changes to make in the make files. Please test from fresh repository copy. Let me know if things like the following work:

make htb
make install-htb
make uninstall-htb
make qhc
make install-qhc
make uninstall-qhc

"make uninstall" should uninstall the help/qt and help/wx subdirectories. However, "make install" does not automatically install help/qt and help/wx.

Give

make html
make html-install

a try. I'm not having much success with that, but I don't think that has anything to do with the patch modifications.

[Karl Ratzsch]

Thanks, works quite well for wxt terminal, help is displayed with images and all. The www links to the demos dont work, but i guess this is expected.

When trying to use a www link, nothing happens, but a warning window appears when closing the gnuplot session afterwards:

14:29:01: Unable to open requested HTML document: http://www.gnuplot.info/demo/fillbetween.html

With qt, i arrived at the correct errormessage about the not-installed help files.

I cannot "make qhc" with qt5, however, it tries to use the qt4 qcollectiongenerator:

ratzsch@Fenchurch:~/src/gnuplot/docs$ make qhc
rm -f qt/gnuplot.qhc qt/gnuplot.qch
cp -ap ./qt/gnuplot.qhcp helpbook
cp -ap qt/gnuplot.qhp helpbook
cd helpbook; qcollectiongenerator gnuplot.qhcp -o gnuplot.qhc; cd ..
qcollectiongenerator: could not exec '/usr/lib/x86_64-linux-gnu/qt4/bin/qcollectiongenerator': No such file or directory
mv -f helpbook/gnuplot.qhc qt
mv: Aufruf von stat für »helpbook/gnuplot.qhc“ nicht möglich: Datei oder Verzeichnis nicht gefunden
Makefile:1200: recipe for target 'qt/gnuplot.qhc' failed
make: *** [qt/gnuplot.qhc] Error 1

I haven't tried installing qt4.

[Dan Sebald]

I have qt4, so you probably shouldn't have problems there. Did the patch used to work with qt5? If so, it's probably a make file configuration issue.

Why wouldn't the http://www.gnuplot.info/demo/fillbetween.html location be found? It seems available. The thing is, I'm not sure I've programmed the external path into the patch, so it might be somewhere else in the code that is coming from. I'll investigate.

[Dan Sebald]

The Qt 5 issue might be something with your installation because I can't find any such path in the build. The name of the program 'qcollectiongenerator' remains the same between v4 and v5.

Here's the closest discussion-list item I can find:

https://musescore.org/en/node/47406

You may need to investigate a bit, like check where Qt5 is located on your system. E.g., why would an executable be in a path that is labelled /usr/lib/x86_64-linux-gnu? Did you install Qt5 with no Qt4 previously installed?

[Karl Ratzsch]

(Back in January, i only tried wxt, sorry. No idea if qt5 worked.)

I'm on ubuntu 15.04, with no qt4 dev-packages installed. /usr/lib/x86_64-linux-gnu/qt5/bin does contain this qcollectiongenerator, along with a few other executables that look development-related (qmake, ...). The same path with ../qt4/bin is mostly empty.

[Dan Sebald]

Oh, so you have the qt4 package installed, but not the qt4 developer packages? When, then qcollectiongenerator is probably in the qt4 dev-packages. I would assume then on your system that the configuration is finding qt4, but not qt5. Do the following in the build directory (may be the same as the source directory):

[sebald@ build6]$ grep qt config.log
PATH: /usr/lib64/qt-3.3/bin/
configure:15868: result: Qt terminal: yes (qt4)
LRELEASE='/usr/lib64/qt4/bin/lrelease'
MOC='/usr/lib64/qt4/bin/moc'
RCC='/usr/lib64/qt4/bin/rcc'
UIC='/usr/lib64/qt4/bin/uic'
config.status:1067: creating src/qtterminal/Makefile

Are you seeing "Qt terminal: yes (qt4)" or qt5? You could either install the qt4 developer package, or maybe remove the qt4 base package and see if gnuplot config then finds qt5. The configuration is supposed to look for qt5 first, so I'm not sure how successful removing qt4 base package will be.

[Karl Ratzsch]

./configure says it's found qt5. I found no reference to qt4 in config.log (attached).

ratzsch@Fenchurch:~/src/gnuplot$ grep qt4 config.log
ratzsch@Fenchurch:~/src/gnuplot$ grep qt config.log
  $ ./configure --with-wx-single-threaded --with-qt --program-suffix=51
config.status:1109: creating src/qtterminal/Makefile
configure:16439: result:   Qt terminal: yes (qt5)
pkg_cv_QT_CFLAGS='-I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtSvg -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtPrintSupport -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtWidgets -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtGui -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtCore -I/usr/include/x86_64-linux-gnu/qt5 '
CPPFLAGS='   -I/usr/local/include -pthread -I/usr/include/pango-1.0 -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12  -pthread -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include  -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtSvg -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtPrintSupport -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtWidgets -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtGui -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtCore -I/usr/include/x86_64-linux-gnu/qt5 '
LRELEASE='/usr/lib/x86_64-linux-gnu/qt5/bin/lrelease'
MOC='/usr/lib/x86_64-linux-gnu/qt5/bin/moc'
QT_CFLAGS='-I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtSvg -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtPrintSupport -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtWidgets -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtGui -I/usr/include/x86_64-linux-gnu/qt5 -I/usr/include/x86_64-linux-gnu/qt5/QtCore -I/usr/include/x86_64-linux-gnu/qt5 '
RCC='/usr/lib/x86_64-linux-gnu/qt5/bin/rcc'
UIC='/usr/lib/x86_64-linux-gnu/qt5/bin/uic'

qt4 is needed by some other program. I could try to find another machine without it if necessary.

If I execute "qcollectiongenerator" on the bash shell, i get the same errormessage. That's a bug in the dependencies of ubuntus qt packages, right? But also "make qhc" should explicitly call the qt5 version if configured for qt5, i guess.

(I now just made a symlink for the qcollectiongenerator, and the qt help viewer looks alright!)

[Dan Sebald]

Good that Qt compiles... Yes, that is some type of ubuntu error, but it is kind of confusing. On my fedora system the "qcollectiongenerator" executable is in /usr/bin while a symbolic link are made to it from /usr/lib64/qt4/bin. It seems with that sort of configuration, having two versions of Qt at the same time means one of them has to be pointing at an incorrect version.

The symbolic link seems the thing to do, but you might want to make sure that your other Qt4 requirement still works properly. An alternative might have been to install the Qt4 developer libraries and then use the "configure" option --with-qt=qt4

  --with-qt =qt4 =qt5 =no Qt terminal (default autodetect)

OK, so that leaves html not quite working correctly, but I think that is a bigger and ancillary issue.

[Karl Ratzsch]

I notice after running "make clean", "make qhc" is broken. and "make distclean" seems to mess up the whole source tree, i had to delete and rebuild it afterwards.

make html is a bit tricky, if it finds the graphs prepared by another target, it makes a (mostly, no graphs) working html documentation in docs/htmldocs. sometimes. at least i got it once. "make install-html" always gave a lot of complaints about graph conversion

Error: Cannot read 'img4.png': Datei oder Verzeichnis nicht gefunden
Converting image #3
pstoimg: Warning: Cannot use /tmp/l2h29275 as temporary directory.
pstoimg: Error: Cannot find file "/tmp/l2h29275/_image003.ps": Datei oder Verzeichnis nicht gefunden

Error while converting image

One stupid question: The qt/wxt help viewer has not yet any connection with the "help" command at the gnuplot prompt, right?

I'll be glad to test new versions of this, just keep me posted.

[Dan Sebald]

OK. I've added some dependencies so that "make qhc" and "make htb" will cause allterm.h and gnuplot executable to be built prior. See attached patch.

Yes, no connection between qt/wxt help viewers and command line "help". I think that would be too much for one patch, and would need some thought to coordinate.

That html build experience sounds like mine. I think that issue too is beyond the scope simply because that behavior may not be due to the patch changes.

[Karl Ratzsch]

Do you also see that gnuplot gives a memory access error when you try to close the gp session with the wxt help viewer still open? I'm using wxt3.0.

[Ethan Merritt]

I'm just looking at this for the first time.
It's not working for me.

wxt:
make htb install-htb appears to work without error
Clicking on the book icon in the wxt gui causes a continuously rotating sand-clock icon to appear; the program consumes 100% CPU and hangs until killed externally

qt:
There is no program qcollectiongenerator on my machines, so I can't try this. It is not in any of the packages obviously related to qt help files:
libqt5help-devel 5.0.2 1.mga3 i586
libqthelp5 5.0.2 1.mga3 i586
or the two support packages these pull in:
libqtclucene5 5.0.2 1.mga3 i586
libqt5sql5 5.0.2 1.1.mga3 i586

[Dan Sebald]

I'm not sure what the wxWidgets/KDE issue is. Did you first attempt to launch the wxWidgets helpbook when it wasn't yet installed? (Try "make uninstall-htb".) Do you see a sandclock then as well? Or is the dialog box appearing that indicates that documentation must be installed (proper behavior)? If the sandclock appears even if the doc isn't installed, then the problem lies earlier in the helpbook process, such as when clicking on the button and attempting to find the documentation (something more likely my fault). If the dialog message appears (as designed) then it is probably later, such as poorly formatted doc html.

On Fedora, I found qcollectiongenerator in the norm qt toolkit developers RPM. It's called "Development files for the Qt toolkit" named "qt-devel-1:4.7.4-7.fc14". The Qt help and help-devel would seem like the logical locations, wouldn't it? Look around for other bundle libraries, I guess.

[Ethan Merritt]

wxt:
Before installing the helpbook clicking on the book widget produced a message saying something like "Help manual is not installed - to install ..."
After installing the helpbook clicking on the widget causes the program to hang.

qt:
The devel package you mention is for qt4. I am running qt5. I have done a content search on the complete set of qt packages without finding qcollectiongenerator (although I don't necessarily believe the search index is exhaustive). Perhaps it is built during installation of some subsystem rather than being provided as a pre-built tool?

[Dan Sebald]

wxt:
OK, so the problem is not locating the compressed HTML files. I suspect it is the DisplayContents() call that is hanging, but we should confirm. I'm attaching a short diff that you can apply to the patched directory tree. It simply prints to stderr some text to locate which function is hanging.

qt:
Ah, Qt5. I didn't think of that. There is a qcollectiongenerator-qt5 file. I would guess that bundles should have a link between qcollectiongenerator and that qcollectiongenerator-qt5 file. (On my system qcollectiongenerator is not a link but the actual executable.) At a terminal window command line, if you type "qcol" and then double-tab does qcollectiongenerator-qt5 appear?

[Ethan Merritt]

It prints the first three messages, then hangs

Terminal type set to 'wxt'
Options are '0 enhanced'
gnuplot> plot sinc(x)
gnuplot>
gnuplot> OnHelp 5
OnHelp 10
OnHelp 15
Killed