From: Arjen M. <arj...@de...> - 2010-12-15 07:50:32
|
Hi Alan, my guess here is that PLplot itself is limiting the file name. I have not had a chance yet to check this theory, but when I used the relative very much shorter path, it all worked fine. (The output from the script shows the entire path, so that is not truncated) Regards, Arjen On 2010-12-07 17:40, Alan W. Irwin wrote: > On 2010-12-07 13:13+0100 Arjen Markus wrote: > >> Hi Alan, >> >> I changed the value of the "results" variable in test_tcl.sh.in >> to "${OUTPUT_DIR}" and it worked fine. Except that the Tcl results >> do not show up in the summary. Odd. > > One other curiosity. If you look higher in test_tcl.sh(.in) you find the > following shell commands: > > cd "${OUTPUT_DIR}" > results="`pwd`" > export results > > So ${results} should normally refer to the same directory as > ${OUTPUT_DIR}. However, the pwd command always returns absolute > results so therefore ${results} is absolute as well, while > ${OUTPUT_DIR} is relative if I recall correctly. So your earlier idea > that ${results} refers to a pathname too long for Windows to handle > _might_ be the explanation of why ${OUTPUT_DIR} works but ${results} does > not. On the other hand, the thread starting at > http://lists-archives.org/mingw-users/18294-msys-bash-maximum-length-of-command-line-arguments.html > > indicates the Windows pathname limit is 8191 characters, and in any > case we are using bash here which that thread implies has even a longer > pathname limit. > > I have no explanation of why the Tcl results are not showing up in the > summary except possibly they are put into the wrong directory. Maybe > the absolute pathname really is needed? I will look further at all of > this once I have downloaded and installed everything I need (including > Tcl) to do a thorough test of PLplot under wine. > > Alan > __________________________ > Alan W. Irwin > > Astronomical research affiliation with Department of Physics and Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS equation-of-state implementation > for stellar interiors (freeeos.sf.net); PLplot scientific plotting software > package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of > Linux Links project (loll.sf.net); and the Linux Brochure Project > (lbproject.sf.net). > __________________________ > > Linux-powered Science > __________________________ > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2010-12-15 20:21:50
|
On 2010-12-15 08:50+0100 Arjen Markus wrote: > Hi Alan, > > my guess here is that PLplot itself is limiting the file name. I have > not had a chance yet to check this theory, but when I used the relative > very much shorter path, it all worked fine. (The output from the script > shows the entire path, so that is not truncated) Hi Arjen: I have just installed the Windows version of Tcl, and for an unmodified svn trunk version examples 1 to 13 execute without problems on wine, but I verify the problem with Tcl example 14 on that (wine) platform. The message I get is Can't open /Z/home/wine/newstart/plplot_comprehensive_test_disposable1/shared/build_tree/examples/x14at.psc I will investigate independently (starting with your hypothesis that PLplot itself is limiting the filename length which seems like a good possibility to me). If you make some further progress on this issue, please let me know, and vice versa, of course. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2010-12-16 07:41:10
|
Hi Alan, okay, at the very least this error is consistent, so that should make it easier to hunt down. I think I have some opportunity tonight to further investigate this and other Windows matters. Will keep you posted. Regards, Arjen On 2010-12-15 21:21, Alan W. Irwin wrote: > On 2010-12-15 08:50+0100 Arjen Markus wrote: > >> Hi Alan, >> >> my guess here is that PLplot itself is limiting the file name. I have >> not had a chance yet to check this theory, but when I used the relative >> very much shorter path, it all worked fine. (The output from the script >> shows the entire path, so that is not truncated) > > Hi Arjen: > > I have just installed the Windows version of Tcl, and for an > unmodified svn trunk version examples 1 to 13 execute without problems > on wine, but I verify the problem with Tcl example 14 on that (wine) > platform. The message I get is > > Can't open > /Z/home/wine/newstart/plplot_comprehensive_test_disposable1/shared/build_tree/examples/x14at.psc > > > I will investigate independently (starting with your hypothesis that > PLplot itself is limiting the filename length which seems like a good > possibility to me). If you make some further progress on this issue, > please let me know, and vice versa, of course. > > Alan > __________________________ > Alan W. Irwin > > Astronomical research affiliation with Department of Physics and Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS equation-of-state implementation > for stellar interiors (freeeos.sf.net); PLplot scientific plotting software > package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of > Linux Links project (loll.sf.net); and the Linux Brochure Project > (lbproject.sf.net). > __________________________ > > Linux-powered Science > __________________________ > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2010-12-16 19:35:31
|
On 2010-12-16 08:40+0100 Arjen Markus wrote: > Hi Alan, > > okay, at the very least this error is consistent, so that should make > it easier to hunt down. I think I have some opportunity tonight to > further investigate this and other Windows matters. Will keep you > posted. Hi Arjen: As in all debugging the essential is to simplify as much as possible while still demonstrating the bug. Here are some steps of that simplification process that I have been able to do so far. It leads to a rather improbable hypothesis.... First run comprehensive_test.sh so that "can't open" error shows up in the comprehensive_test.sh build tree, /z/home/wine/newstart/plplot_comprehensive_test_disposable1/shared/build_tree. If after that bad wine result, I set all appropriate environment variables, cd to examples/tcl in the above tree then run echo $(pwd)/test1.ps |pltcl x14 -dev psc -o test.ps Demo of multiple output streams via the psc driver. Running with the second stream as slave to the first. Enter graphics output file name: Can't open /z/home/wine/newstart/plplot_comprehensive_test_disposable1/shared/build_tree/examples/tcl/test1.ps. .... I then tried the same thing in the corresponding c example, and got the same (bad) result. I then tried ./x10c -dev psc -o $(pwd)/test.ps and echo test1.ps| ./x14c -dev psc -o $(pwd)/test.ps and the long filename specified with -o worked fine! I then tried interactive input of the long file name, and that generated the error. ./x10c -dev psc Enter graphics output file name: /z/home/wine/newstart/plplot_comprehensive_test_disposable1/shared/build_tree/examples/c/test.ps Can't open /z/home/wine/newstart/plplot_comprehensive_test_disposable1/shared/build_tree/examples/c/test.ps. ... This is the simplest instance of the error I have been able to generate so far. I then tried many of the same examples under Linux and could not reproduce the error at all no matter how long I made the full path to the file, e.g., echo $(pwd)/../../../../shared/build_tree/examples/c/test1.ps /home/software/plplot_svn/HEAD/plplot_cmake_qt/comprehensive_test_disposeable/shared/build_tree/examples/c/../../../../shared/build_tree/examples/c/test1.ps The bad C results on wine show the error has nothing to do with Tcl which is a nice simplification, but the corresponding good results on Linux show it must be some Windows-specific issue. On the other hand, the good C results using the -o option on wine prove bash and C and PLplot can handle long filenames on wine with no issues. Thus, all the symptoms are consistent with the hypothesis that for the special case where the user must be prompted for a filename, the C (and Tcl) examples call a PLplot function to open the file in a peculiar way which has a Windows-specific filename length limitation. That hypothesis does seem pretty improbable if you think about it, but as Sherlock Holmes said: "Eliminate all other factors, and the one which remains must be the truth." That should be enough experimentation, and it is now time to start debugging the code to see whether the above hypothesis is true. Fortunately, MSYS gives you access to the gcc debugger, gdb, which is an excellent tool for debugging. More later as this detective story unfolds. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Alan W. I. <ir...@be...> - 2010-12-17 07:28:32
|
On 2010-12-16 11:35-0800 Alan W. Irwin wrote: > The bad C results on wine show the error has nothing to do with Tcl > which is a nice simplification, but the corresponding good results on > Linux show it must be some Windows-specific issue. On the other hand, > the good C results using the -o option on wine prove bash and C and > PLplot can handle long filenames on wine with no issues. > > Thus, all the symptoms are consistent with the hypothesis that for the > special case where the user must be prompted for a filename, the C > (and Tcl) examples call a PLplot function to open the file in a > peculiar way which has a Windows-specific filename length limitation. > That hypothesis does seem pretty improbable if you think about > it, but as Sherlock Holmes said: "Eliminate all other factors, and the > one which remains must be the truth." > > That should be enough experimentation, and it is now time to start > debugging the code to see whether the above hypothesis is true. Fortunately, > MSYS gives you access to the gcc debugger, gdb, which is an excellent > tool for debugging. Hmm. Holmesian logic only works if you truly have eliminated all other possibilities, and this was not the case here. It wasn't a question of the length of the pathname. Instead, it was a question of whether the pathname was relative or absolute. For the case of the prompted filename that was absolute, the drive letter form is fine, but not the alternative (available under bash) of the leading-slash form of absolute filename. For example, echo z:/tmp/test.ps |./x10c -dev psc works without issues, but using the /z prefix fails: echo /z/tmp/test.ps |./x10c -dev psc Enter graphics output file name: Can't open /z/tmp/test.ps. Can't open /z/tmp/test.ps..... Furthermore, ./x10c -dev psc -o /z/tmp/test.ps _does_ work despite that /z prefix, and if you use the -debug option, you find it successfully opens z:/tmp/test.ps rather than the specified /z/tmp/test.ps! However, if you try to use that -o option in a gdb environment, e.g., gdb ./x10c run -dev psc -o /z/tmp/test.ps it fails with the "Can't open /z/tmp/test.ps" message. So there are some really subtle interactions going on here in a bash/gdb/windows environment which provide a wide variety of failing or succeeding results depending on minute details of exactly what is done with absolute pathnames and how various software components recognize the leading / form (or not in the gdb case). In all cases the drive letter form works, but I didn't want to complicate our C code to transform the leading slash form to drive letter form for the Windows case. So I have worked around this issue (revision 11374) by using a sed script (sed is available for MSYS) to transform absolute path names to the drive-letter form. The result works on MinGW/MSYS/wine for the first time! Here are the diff summary results on that platform. tcl Missing examples : 33 Differing postscript output : 04 17 18 24 26 29 Missing stdout : Differing stdout : The 33, 04, 18, and 26 differences also occur on Linux and I presume they are all due to the remaining plstring, plstring3, and pllegend propagation issues to the Tcl examples (and possibly propagation issues to the Tcl bindings for pllegend). That leaves the differences for examples 17, 24, and 29 that occur on 32-bit wine, but not 64-bit Linux to investigate. diff reveals many example 17 differences between Tcl and C, but I cannot see any visual differences in the final results viewed with gv. diff reveals only ~30 or so lines of example 24 differences between Tcl and C, but there are obvious visual differences (extra characters in the Tcl version) in the results viewed with gv. ndiff --abserr 2 shows only two numerical differences for example 29 between Tcl and C that are larger than 2 in the last place, but those two were quite large. gv showed small visual differences for the plot of the time discontinuities for pages 5 and 6 that presumably reflect those two large differences in the ndiff results. I ascribe this issue to rounding errors in the exact x position of the discontinuties. The problem will likely disappear if we move the discontinuties slightly away from page centre. In sum, probably example 17 and 24 need some more work on Tcl for 32-bit Windows platforms (and possibly 32-bit Linux platforms) while a change of all examples 29 to shift the discontinuties slightly from the exact centre of the plot will probably solve that (32-bit Tcl?) issue. Aside from these rendering issues it looks like everything works at least on my wine version of the Windows platform, and that good result needs confirmation on a Microsoft version of the Windows platform as well. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2010-12-17 08:24:12
|
Hi Alan, that is wonderful! My guess as to the subtleties of the two forms of paths is that bash (or any shell) under MinGW/MSYS recognises the /z/path/to/file form and _converts_ it to z:\path\to\file (or one with /'s), to make sure that ordinary programs that assume the DOS form of absolute paths continue to work (forward slashes usually work as well!). I just checked it with a small program: /* chkpath.c -- Check the path as provided by bash */ #include <stdio.h> int main(int argc, char *argv[] ) { printf( "Argument 0: %s\n", argv[0] ); printf( "Argument 1: %s\n", argv[1] ); } The result of "chkpath /a/b" was: Argument 0: d:\plplot-svn\test-drvdir\chkpath.exe Argument 1: d:/MinGW2010/msys/1.0/a/b Whereas when I run "chkpath a:/a/b" the result is: Argument 0: d:\plplot-svn\test-drvdir\chkpath.exe Argument 1: a:/a/b If I run "chkpath /d/b" (so with a drive letter that corresponds to an actual drive) I get: Argument 0: d:\plplot-svn\test-drvdir\chkpath.exe Argument 1: d:/b Conclusion: the shell is performing all manner of magical tricks to make sure the UNIX and Windows conventions both work! Regards, Arjen On 2010-12-17 08:28, Alan W. Irwin wrote: > On 2010-12-16 11:35-0800 Alan W. Irwin wrote: > >> The bad C results on wine show the error has nothing to do with Tcl >> which is a nice simplification, but the corresponding good results on >> Linux show it must be some Windows-specific issue. On the other hand, >> the good C results using the -o option on wine prove bash and C and >> PLplot can handle long filenames on wine with no issues. >> >> Thus, all the symptoms are consistent with the hypothesis that for the >> special case where the user must be prompted for a filename, the C >> (and Tcl) examples call a PLplot function to open the file in a >> peculiar way which has a Windows-specific filename length limitation. >> That hypothesis does seem pretty improbable if you think about >> it, but as Sherlock Holmes said: "Eliminate all other factors, and the >> one which remains must be the truth." >> >> That should be enough experimentation, and it is now time to start >> debugging the code to see whether the above hypothesis is true. >> Fortunately, >> MSYS gives you access to the gcc debugger, gdb, which is an excellent >> tool for debugging. > > Hmm. Holmesian logic only works if you truly have eliminated all > other possibilities, and this was not the case here. It wasn't a > question of the length of the pathname. Instead, it was a question > of whether the pathname was relative or absolute. For the case > of the prompted filename that was absolute, the drive letter > form is fine, but not the alternative (available under bash) of > the leading-slash form of absolute filename. > > For example, > > echo z:/tmp/test.ps |./x10c -dev psc > > works without issues, but using the /z prefix fails: > > echo /z/tmp/test.ps |./x10c -dev psc > Enter graphics output file name: Can't open /z/tmp/test.ps. > Can't open /z/tmp/test.ps..... > > Furthermore, > > ./x10c -dev psc -o /z/tmp/test.ps > > _does_ work despite that /z prefix, and if you use the -debug option, you > find it successfully opens z:/tmp/test.ps rather than the specified > /z/tmp/test.ps! However, if you try to use that -o option in a gdb > environment, e.g., > > gdb ./x10c > run -dev psc -o /z/tmp/test.ps > > it fails with the "Can't open /z/tmp/test.ps" message. So there are > some really subtle interactions going on here in a bash/gdb/windows > environment which provide a wide variety of failing or succeeding > results depending on minute details of exactly what is done with > absolute pathnames and how various software components recognize > the leading / form (or not in the gdb case). > > In all cases the drive letter form works, but I didn't want to > complicate our C code to transform the leading slash form to drive > letter form for the Windows case. So I have worked around this issue > (revision 11374) by using a sed script (sed is available for MSYS) to > transform absolute path names to the drive-letter form. The result > works on MinGW/MSYS/wine for the first time! > > Here are the diff summary results on that platform. > > tcl > Missing examples : 33 > Differing postscript output : 04 17 18 24 26 29 > Missing stdout : > Differing stdout : > > > The 33, 04, 18, and 26 differences also occur on Linux and I presume > they are all due to the remaining plstring, plstring3, and pllegend > propagation issues to the Tcl examples (and possibly propagation > issues to the Tcl bindings for pllegend). > > That leaves the differences for examples 17, 24, and 29 that occur > on 32-bit wine, but not 64-bit Linux to investigate. > > diff reveals many example 17 differences between Tcl and C, but I > cannot see any visual differences in the final results viewed > with gv. > > diff reveals only ~30 or so lines of example 24 differences between Tcl and > C, but there are obvious visual differences (extra characters in the Tcl > version) in the results viewed with gv. > > ndiff --abserr 2 shows only two numerical differences for example 29 > between Tcl and C that are larger than 2 in the last place, but those > two were quite large. gv showed small visual differences for the plot > of the time discontinuities for pages 5 and 6 that presumably reflect > those two large differences in the ndiff results. I ascribe this > issue to rounding errors in the exact x position of the > discontinuties. The problem will likely disappear if we move the > discontinuties slightly away from page centre. > > In sum, probably example 17 and 24 need some more work on Tcl for > 32-bit Windows platforms (and possibly 32-bit Linux platforms) while a > change of all examples 29 to shift the discontinuties slightly from > the exact centre of the plot will probably solve that (32-bit Tcl?) > issue. Aside from these rendering issues it looks like everything > works at least on my wine version of the Windows platform, and that > good result needs confirmation on a Microsoft version of the Windows > platform as well. > > Alan > __________________________ > Alan W. Irwin > > Astronomical research affiliation with Department of Physics and Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS equation-of-state implementation > for stellar interiors (freeeos.sf.net); PLplot scientific plotting software > package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of > Linux Links project (loll.sf.net); and the Linux Brochure Project > (lbproject.sf.net). > __________________________ > > Linux-powered Science > __________________________ > > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Arjen M. <arj...@de...> - 2010-12-17 10:13:42
|
Hi Alan, On 2010-12-17 08:28, Alan W. Irwin wrote: > In all cases the drive letter form works, but I didn't want to > complicate our C code to transform the leading slash form to drive > letter form for the Windows case. So I have worked around this issue > (revision 11374) by using a sed script (sed is available for MSYS) to > transform absolute path names to the drive-letter form. The result > works on MinGW/MSYS/wine for the first time! > > Here are the diff summary results on that platform. > > tcl > Missing examples : 33 > Differing postscript output : 04 17 18 24 26 29 > Missing stdout : > Differing stdout : > > > The 33, 04, 18, and 26 differences also occur on Linux and I presume > they are all due to the remaining plstring, plstring3, and pllegend > propagation issues to the Tcl examples (and possibly propagation > issues to the Tcl bindings for pllegend). > > That leaves the differences for examples 17, 24, and 29 that occur > on 32-bit wine, but not 64-bit Linux to investigate. > Odd, on Windows with MinGW/MSYS I see differences with 04, 18, 24 and 26 only (and 33 is missing). Example 24 is the odd one out - the others also occur for all other languages. This deserves some digging into ... Regards, Arjen DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2010-12-18 19:06:23
|
I should have also said that PLplot users on the Windows platform can find directions for installing a Windows version of Tcl at http://www.miscdebris.net/plplot_wiki/index.php?title=Install_Tcl which is linked from http://www.miscdebris.net/plplot_wiki/index.php?title=Specifics_for_various_platforms Of course, that Install_Tcl page was writen from my own wine Windows perspective so Microsoft Windows users are requested to modify that page to add the Microsoft Windows perspective as well. And similarly for the MinGW/MSYS, SWIG, Python, and Lua pages I have recently created that are linked from the "Specifics_for_various_platforms" page. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2010-12-20 09:00:51
|
Hi Alan, On 2010-12-17 08:28, Alan W. Irwin wrote: > In all cases the drive letter form works, but I didn't want to > complicate our C code to transform the leading slash form to drive > letter form for the Windows case. So I have worked around this issue > (revision 11374) by using a sed script (sed is available for MSYS) to > transform absolute path names to the drive-letter form. The result > works on MinGW/MSYS/wine for the first time! > > Here are the diff summary results on that platform. > > tcl > Missing examples : 33 > Differing postscript output : 04 17 18 24 26 29 > Missing stdout : > Differing stdout : > > > The 33, 04, 18, and 26 differences also occur on Linux and I presume > they are all due to the remaining plstring, plstring3, and pllegend > propagation issues to the Tcl examples (and possibly propagation > issues to the Tcl bindings for pllegend). > > That leaves the differences for examples 17, 24, and 29 that occur > on 32-bit wine, but not 64-bit Linux to investigate. > > diff reveals many example 17 differences between Tcl and C, but I > cannot see any visual differences in the final results viewed > with gv. > > diff reveals only ~30 or so lines of example 24 differences between Tcl and > C, but there are obvious visual differences (extra characters in the Tcl > version) in the results viewed with gv. > I think the issue has to do with the character encodings. Tcl internally works with UTF-8 and on Windows the system encoding is cp1252 (at least on my machine). I have been experimenting with different encodings but no success yet. I guess I first have to understand _exactly_ what is going on before I can solve it. I do not see differences with examples 17 and 29 on plain Windows, though. > > In sum, probably example 17 and 24 need some more work on Tcl for > 32-bit Windows platforms (and possibly 32-bit Linux platforms) while a > change of all examples 29 to shift the discontinuties slightly from > the exact centre of the plot will probably solve that (32-bit Tcl?) > issue. Aside from these rendering issues it looks like everything > works at least on my wine version of the Windows platform, and that > good result needs confirmation on a Microsoft version of the Windows > platform as well. Example 17's differences are a bit puzzling. I have no clue yet. I can reproduce differences with example 29, so that is easier. As for my test results with F77 example 9: it turns out that on the DLL side the array "tr" as stored in COMMON block /plplot/ is completely zero. That means the transformed coordinates are (0,0) for all points. The question _why_ it turns out to be zero is intriguing, but likely a nasty problem with data in DLLs on Windows. I know a workaround for this (an extra subroutine to fill the array), but I want to check that this is indeed the cause first. Regards, Arjen DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Arjen M. <arj...@de...> - 2010-12-20 13:31:45
|
Hello all, On 2010-12-20 10:00, Arjen Markus wrote: > > As for my test results with F77 example 9: it turns out that on the > DLL side the array "tr" as stored in COMMON block /plplot/ is completely > zero. That means the transformed coordinates are (0,0) for all points. > The question _why_ it turns out to be zero is intriguing, but likely > a nasty problem with data in DLLs on Windows. I know a workaround for > this (an extra subroutine to fill the array), but I want to check that > this is indeed the cause first. > I can confirm that this is an issue on Windows (for the F90 bindings we use a different approach). I have posed a question about it on the gfortran mailing list. Regards, Arjen DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Arjen M. <arj...@de...> - 2010-12-21 08:36:57
|
Hello all, On 2010-12-20 14:31, Arjen Markus wrote: > Hello all, > > On 2010-12-20 10:00, Arjen Markus wrote: > >> As for my test results with F77 example 9: it turns out that on the >> DLL side the array "tr" as stored in COMMON block /plplot/ is completely >> zero. That means the transformed coordinates are (0,0) for all points. >> The question _why_ it turns out to be zero is intriguing, but likely >> a nasty problem with data in DLLs on Windows. I know a workaround for >> this (an extra subroutine to fill the array), but I want to check that >> this is indeed the cause first. >> > > I can confirm that this is an issue on Windows (for the F90 bindings we > use a different approach). I have posed a question about it on the > gfortran mailing list. > I got a suggestion for this problem, but it did not work the way it was supposed to. I will keep looking. Regards, Arjen DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Arjen M. <arj...@de...> - 2010-12-21 11:30:21
|
Hello, here is an update on the FORTRAN 77 issue: The proposed solution (a compiler directive) does not work with the current gfortran compiler (issue http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47030). I can think of several workarounds: - Expand the argument list of plcon1 with the array tr This breaks existing code though - Use a separate subroutine to fill the array in the COMMON block - Use a MODULE despite the fact that we are talking about FORTRAN 77 - all maintained Fortran compilers that I know of are at least conforming to the Fortran 90 standard that introduced modules. - Put the implementation of plcon1 in the same - static - library as plparseopts (which had a very similar issue) The second solution is probably the easiest for us developers, the last the most convenient for us users. Regards, Arjen On 2010-12-21 09:36, Arjen Markus wrote: > Hello all, > > On 2010-12-20 14:31, Arjen Markus wrote: >> Hello all, >> >> On 2010-12-20 10:00, Arjen Markus wrote: >> >>> As for my test results with F77 example 9: it turns out that on the >>> DLL side the array "tr" as stored in COMMON block /plplot/ is completely >>> zero. That means the transformed coordinates are (0,0) for all points. >>> The question _why_ it turns out to be zero is intriguing, but likely >>> a nasty problem with data in DLLs on Windows. I know a workaround for >>> this (an extra subroutine to fill the array), but I want to check that >>> this is indeed the cause first. >>> >> >> I can confirm that this is an issue on Windows (for the F90 bindings we >> use a different approach). I have posed a question about it on the >> gfortran mailing list. >> > > I got a suggestion for this problem, but it did not work the way it > was supposed to. I will keep looking. > > Regards, > > Arjen > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2010-12-12 06:58:27
|
On 2010-12-06 11:53-0800 Alan W. Irwin wrote: > [...] The "C example 29 is missing" message above is a left over > from when we excluded example 29 in the Windows case because we didn't > have time functionality that was suitable for that platform. But now > example 29 should work fine on Windows [...] [by] > modifying the critical_examples variable in > plplot_test/plplot-test.sh.cmake Done as of revision 11368. The example 29 result looks fine on wine (IOW, libqsastime is doing the job on Windows platforms). Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2010-12-30 08:07:50
|
Hello, I have found an easy way to solve the UTF-8 problem, source -econding utf-8 (not cp1252). But it works only for Tcl 8.5, as the [source] command was extended with that option then. I will either have to see about an alternative or make sure this option is not used for Tcl 8.4 and older. Regards, Arjen On 2010-12-28 11:10, Arjen Markus wrote: > Hi Alan, > > Tcl has two options to do what you suggest: > - Set the system encoding to cp1252 > - Source the file using -encoding cp1252 > > (A third is to adapt the C side of the Plplot API.) > > I will experiment with both. > > Regards, > > Arjen > > On 2010-12-24 22:20, Alan W. Irwin wrote: >> On 2010-12-24 13:05+0100 Arjen Markus wrote: >> >>> Hi Alan, >>> >>> On 2010-12-21 20:03, Alan W. Irwin wrote: >>> >>>> >>>> I will stop now and not comment more on the Tcl case, because I think >>>> it is essential to focus on C for now and one of the cairo or qt >>>> devices. >>>> >>>> Of course, as I have stated before the psc device driver is not useful >>>> for diagnosis of encoding issues because everything exotic such as the >>>> Mandarin Peace word ends up as blanks in any case because the standard >>>> Type 1 fonts that the psc device uses have an extremely limited glyph >>>> set that does not include Mandarin glyphs or any other non-English >>>> glyphs besides Greek (for mathematical purposes). >>>> >>> >>> I have checked the C and F95 examples 24 with the wxWidgets device on >>> Windows: they work very nicely, except that my PC does not have all the >>> required fonts. With Tcl I recognise the odd sequences I also see in >>> the source code (viewed from a cp1252 perspective). >> >> Thus, it appears from your results that C and Fortran on Windows >> simply accept the byte sequences mentioned in strings without >> molesting them while your hypothesis is that Tcl does not follow that >> simple model. Instead it does an implicit transformation of all >> strings from (presumed) system encoding to UTF-8 which messes up the >> byte sequences, and your proposed cure is to take all strings input to >> PLplot from Tcl and do the inverse transformation which uses >> a call to Tcl_UtfToExternalDString with a >> NULL encoding. From the man page, what that will do is to convert a >> (presumed) UTF-8 string to the (presumed) system encoding. >> >> If your hypothesis is correct, then your proposed cure might indeed >> work on all platforms. Certainly on Linux, the implicit >> transformation is UTF-8 to UTF-8 or the identity transform (which is >> why Tcl works right now for example 24 on Linux), and your inverse >> transformation would also be the identify transformation on Linux and >> should therefore also work on that platform. >> >> However, I am concerned with the following issues. >> >> 1. All PLplot API arguments that are strings are assumed to be in >> UTF-8. Thus, the call to Tcl_UtfToExternalDString with NULL has to be >> made in the Tcl bindings for _every_ function in the PLplot API that >> has an input string. >> >> 2. Does the implicit transformation work for arbitrary UTF-8 (e.g., >> arbitrary series of 8-bit bytes) or are there some 8-bit bytes which >> cannot be validly interpreted as cp1252 or which have special >> meanings. >> >> 3. Is Tcl_UtfToExternalDString with NULL encoding the exact inverse >> of the implicit transformation? >> >> All of these issues can be dealt with. Obviously some care in the Tcl >> bindings should take care of issue 1 and to alleviate concern about >> issues 2 and 3 completely, it would be a good idea to put together a >> complete test of all 256 possible 8-bit character combinations. I am >> thinking along the lines of generating a file from C with a string of >> all possibilities from 255 down to a zero (to terminate the string). >> Then using an editor copy that exact string of 256 bytes to Tcl source >> code that automatically puts that string through the implicit >> transformation. Then use the Tcl_UtfToExternalDString with NULL to >> transform that string before calling a C programme that simply outputs >> the string. Then compare that output file with the original file with >> 256 characters to see if you get all 256 characters back in their >> original form. >> >> However, to avoid this work it would be better to convince Tcl not to >> do the implicit string transformation in the first place. The way >> this is handled in Python is to put the following string in the >> first or second line of the Python script that identifies the >> whole Python source file is encoded in utf-8: >> >> # -*- coding: utf-8 -*- >> >> We do that for the following Python examples: >> >> software@raven> grep coding: examples/python/xw??.py >> examples/python/xw18.py:# -*- coding: utf-8; -*- >> examples/python/xw24.py:# -*- coding: utf-8; -*- >> examples/python/xw26.py:# -*- coding: utf-8; -*- >> examples/python/xw33.py:# -*- coding: utf-8; -*- >> >> I think Tcl may have something equivalent in the >> >> encoding system utf-8 >> >> command. From the documentation users are discouraged from using that >> command because it affects everything such as system calls. For >> example, puts would output strings in UTF-8 encoding rather than the >> actual (e.g., cp1252 on your platform) system encoding on Windows >> machines. But is that actually an issue for the above examples? >> First, we don't interact with the operating system (e.g, with puts) as >> far as I know with those examples, and UTF-8 and cp1252 coincide in >> any case for ascii strings. Anyhow, if "encoding system utf-8" works >> for those examples, I think we should use it rather than the more >> difficult steps outlined above. Of course, we should inform Tcl users >> browsing our example code via a comment in those examples that PLplot >> requires utf-8 system encoding for all non-ascii input strings. >> >>> >>> Over the weekend I won't be able to do anything - holiday >>> obligations :). >>> >>> I wish you all a merry Christmas and a happy New Year. >> >> I wish everybody here a "Merry Christmas" and "Happy New Year" as well. >> >> Enjoy your holidays, Arjen, and I hope when you come back you will >> find the "encoding system utf-8" solution works without issues for the >> affected examples. >> >> Alan >> __________________________ >> Alan W. Irwin >> >> Astronomical research affiliation with Department of Physics and >> Astronomy, >> University of Victoria (astrowww.phys.uvic.ca). >> >> Programming affiliations with the FreeEOS equation-of-state >> implementation >> for stellar interiors (freeeos.sf.net); PLplot scientific plotting >> software >> package (plplot.org); the libLASi project (unifont.org/lasi); the >> Loads of >> Linux Links project (loll.sf.net); and the Linux Brochure Project >> (lbproject.sf.net). >> __________________________ >> >> Linux-powered Science >> __________________________ >> > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2011-01-04 19:57:11
|
On 2010-12-30 09:07+0100 Arjen Markus wrote: > Hello, > > I have found an easy way to solve the UTF-8 problem, > source -econding utf-8 (not cp1252). But it works > only for Tcl 8.5, as the [source] command was extended > with that option then. Hi Arjen: It appears from what you have said that Tcl 8.4 is not really ready for UTF-8. Thus, I don't think our Tcl-8.4 users should expect UTF-8 support from us so I suggest we simply ignore UTF-8 for the Tcl-8.4 case. As time goes by that should be less and less of a PLplot limitation as everybody moves to Tcl-8.5. So using that philosophy here is what I propose we do to put this issue to rest. 1. Please test your proposed change under Tcl-8.5 using the test_noninteractive target. I assume you will modify the source statements in examples/tcl/x2[46] and also the source statements in examples/tcl/standard_examples.in and examples/tk/standard_examples.in. The test_noninteractive target only tests examples/tcl/x??, but if the -encoding flag for the source command works in that case I assume it will also work in the standard_examples(.in) Tcl and Tk cases (for Tcl-8.5). 2. Commit your changes. 3. Configure test_tcl.sh(.in) so it skips the unicode examples for Tcl versions less than 8.5. Also, configure examples/tcl/standard_examples.in and examples/tk/standard_examples.in to skip the unicode examples and also drop the -encoding flag on the source command. I can handle all these configuration issues. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2011-01-05 07:36:08
|
Hi Alan, that sounds like a good plan. An alternative would be to use hexadecimal escape codes (\u...) instead of non-ASCII7 characters in the source code, but that has the drawback that you can not view the UNICODE character anymore. I will work on this - earliest commit time will be tomorrow though. Regards, Arjen On Tue, 4 Jan 2011 11:57:01 -0800 (PST) "Alan W. Irwin" <ir...@be...> wrote: > On 2010-12-30 09:07+0100 Arjen Markus wrote: > >> Hello, >> >> I have found an easy way to solve the UTF-8 problem, >> source -econding utf-8 (not cp1252). But it works >> only for Tcl 8.5, as the [source] command was extended >> with that option then. > > Hi Arjen: > > It appears from what you have said that Tcl 8.4 is not >really ready > for UTF-8. Thus, I don't think our Tcl-8.4 users should >expect UTF-8 > support from us so I suggest we simply ignore UTF-8 for >the Tcl-8.4 > case. As time goes by that should be less and less of a >PLplot > limitation as everybody moves to Tcl-8.5. > > So using that philosophy here is what I propose we do to >put this > issue to rest. > > 1. Please test your proposed change under Tcl-8.5 using >the > test_noninteractive target. I assume you will modify the >source > statements in examples/tcl/x2[46] and also the source >statements in > examples/tcl/standard_examples.in and > examples/tk/standard_examples.in. The >test_noninteractive target only > tests examples/tcl/x??, but if the -encoding flag for >the source > command works in that case I assume it will also work in >the > standard_examples(.in) Tcl and Tk cases (for Tcl-8.5). > > 2. Commit your changes. > > 3. Configure test_tcl.sh(.in) so it skips the unicode >examples for Tcl > versions less than 8.5. Also, configure > examples/tcl/standard_examples.in and >examples/tk/standard_examples.in > to skip the unicode examples and also drop the -encoding >flag on the > source command. I can handle all these configuration >issues. > > Alan > __________________________ > Alan W. Irwin > > Astronomical research affiliation with Department of >Physics and Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS >equation-of-state implementation > for stellar interiors (freeeos.sf.net); PLplot >scientific plotting software > package (plplot.org); the libLASi project >(unifont.org/lasi); the Loads of > Linux Links project (loll.sf.net); and the Linux >Brochure Project > (lbproject.sf.net). > __________________________ > > Linux-powered Science > __________________________ > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2010-12-09 09:23:59
|
Hi Arjen: Using revision 11364 of comprehensive_test.sh with all tests excluded other than the test_noninteractive target for the build tree with shared libraries, a minimalist (just C, C++, and Fortran 77 and a few device drivers like ps with no external dependencies) _parallel_ build now works for "MSYS Makefiles" on the wine platform. The diff results were C example 29 is missing c++ Missing examples : 33 Differing postscript output : 04 18 26 Missing stdout : Differing stdout : C example 29 is missing f77 Missing examples : 33 Differing postscript output : 04 09 14a 18 26 Missing stdout : Differing stdout : So I confirm those "extra" 09 and 14a f77 problems on the wine platform that you discovered on the Microsoft Windows platform. Good luck sorting them out. :-) I also confirm a Windows caveat you found for Ada. The Ada bindings fail to create a libplplotadad.dll.a in the dll subdirectory (or anywhere else in the build tree for that matter) so the Ada examples cannot link properly. I have no clue how to modify the Ada language support files to fix this so I am not going to investigate further for now. Some additional caveats that are specific to wine (1) I had to disable Tcl for now because cmake on wine was somehow finding the Linux version of Tcl and bombing out because of that. The eventual solution will be to download and install Tcl for Windows and force cmake to find that correct version. (2) I had to disable Fortran 95 because of a special wine platform problem with Fortran module files. (When parsing the module file it found a parenthesis error which did not exist if you actually looked at the module file). Since you don't have this problem on your Microsoft Windows platform for the exact same MinGW/MSYS versions I use on my wine platform this issue is likely due to some bug in wine's emulation of Windows. I am not going to worry about it and hopefully some wine fix of the future will straighten this out. I also tried the test_noninteractive target in the installed examples tree with the CMake build system, and it silently failed. I confirmed that by hand by running c/x01c in the installed examples build tree. That example returned immediately with no output at all. Assuming you can confirm this issue by hand for the installed examples that are built by cmake (or by invoking comprehensive_test.sh with the --do_test_install_tree yes option), could you investigate further? My guess is it is some PATH issue so the library is not found. I also tried the "MinGW Makefiles" generator. It appears the revisions I put into the comprehensive_test.sh script work fine for this case. The results also _apparently_ showed the test_noninteractive target was built with no errors at all. However, somehow the dependencies were silently dropped for the C++ and Fortran 77 examples. Thus, those examples were not built or run and no diffs were done at all. That is quite a peculiar result. I will be interested to see if you get a similar bad result on your MS Windows platform or whether this is an issue peculiar to the wine platform. Good luck with your use of the revised comprehensive_test.sh script. Meanwhile, I am going to attempt to expand what I test on the wine platform with a bunch of downloads and installs from the web locations I noted in a previous off-list e-mail to you. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Alan W. I. <ir...@be...> - 2010-12-10 21:46:40
|
On 2010-12-09 01:23-0800 Alan W. Irwin wrote: > Using revision 11364 of comprehensive_test.sh with all tests excluded > other than the test_noninteractive target for the build tree with > shared libraries, a minimalist (just C, C++, and Fortran 77 and a few > device drivers like ps with no external dependencies) _parallel_ build > now works for "MSYS Makefiles" on the wine platform. The diff results > were > > C example 29 is missing > c++ > Missing examples : 33 > Differing postscript output : 04 18 26 > Missing stdout : > Differing stdout : > C example 29 is missing > f77 > Missing examples : 33 > Differing postscript output : 04 09 14a 18 26 > Missing stdout : > Differing stdout : > > [...]I also tried the test_noninteractive target in the installed examples > tree with the CMake build system, and it silently failed. Bash on Windows can accept path names in either the drive-letter form (e.g., z:/home/wine/bin) or leading-"/" form (e.g., /z/home/wine/bin). The PATH variable for bash is colon-separated so on windows there is obviously a special interpretation problem for PATH components in the drive-letter form. The symptom of the problem was for the PATH set up by our test scripts which libplplot.dll found that dll without troubles, but executables that needed that dll could not find it because there was an unrelated drive-letter form of component high in the path that confused it. The solution is to make sure all bash scripts set the PATH using components that have been transformed from the drive-letter form to the leading-"/" form (see revision 11366). The result of this revision is comprehensive_test.sh works for the very first time with option --do_test_install_tree yes for MinGW/MSYS on wine Windows. For that case, I get no errors and the same diff report for the installed examples tree as the above diff report for the --do_test_build_tree yes case. That is an extremly promising Windows breakthrough for the installed examples tree case which I plan to exploit by downloading and installing a lot more Window's forms of PLplot dependencies to strengthen the PLplot testing I can do on the wine Windows platform. More later. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Alan W. I. <ir...@be...> - 2010-12-20 19:22:46
|
Hi Arjen: On 2010-12-20 10:00+0100 Arjen Markus wrote: > I think the [example 24] issue has to do with the character encodings. Tcl internally > works with UTF-8 and on Windows the system encoding is cp1252 (at least > on my machine). I have been experimenting with different encodings but > no success yet. I guess I first have to understand _exactly_ what is > going on before I can solve it. To review how this is supposed to work, the language examples (Tcl in this case) should encode each PLplot user string as a UTF-8 string, and that string should be passed directly without transformation to the PLplot C level as a UTF-8 string by the bindings (Tcl in this case). The PLplot C core library then translates the UTF-8 string into an unsigned 32-bit (PLUNICODE) array consisting of FCI's (representing the ascii font-changing commands embedded in the string) and UCS4 indices (representing the UTF-8 unicode encoding in the string). Our modern device drivers such as cairo and qt then interpret that array further to make the appropriate library call for an FCI to change the font or to use the UCS4 index to access a particular glyph for TrueType fonts. There is one more layer of indirection for -dev psc; it translates the UCS4 indices into indices that are suitable for indexing Type 1 fonts with the overwhelming majority of results being the index for Type 1 blank because Type 1 fonts have an extremely limited number of glyphs they can represent. So if there was some screwup in encoding at the Tcl level that generates random UCS4 indices, the most probable response for -dev psc would be a blank glyph which would be rather useless for diagnostic purposes. So I suggest it would be much better to use one of the cairo or qt devices to diagnose what is going wrong at the Tcl example 24 or bindings level since the result of random UCS4 garbage would be more likely to end up as non-blank glyphs. > > I do not see differences with examples 17 and 29 on plain Windows, > though. Is this a 32-bit or 64-bit result? > >> >> In sum, probably example 17 and 24 need some more work on Tcl for >> 32-bit Windows platforms (and possibly 32-bit Linux platforms) while a >> change of all examples 29 to shift the discontinuties slightly from >> the exact centre of the plot will probably solve that (32-bit Tcl?) >> issue. Aside from these rendering issues it looks like everything >> works at least on my wine version of the Windows platform, and that >> good result needs confirmation on a Microsoft version of the Windows >> platform as well. > > Example 17's differences are a bit puzzling. I have no clue yet. > I can reproduce differences with example 29, so that is easier. Can you clarify that last sentence which is in apparent contradiction with what you said about no differences for example 29 above? Thanks for continuing to work away at these Windows issues. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2010-12-21 08:25:28
|
Hi Alan, On 2010-12-20 20:22, Alan W. Irwin wrote: > Hi Arjen: > > On 2010-12-20 10:00+0100 Arjen Markus wrote: > >> I think the [example 24] issue has to do with the character encodings. >> Tcl internally >> works with UTF-8 and on Windows the system encoding is cp1252 (at least >> on my machine). I have been experimenting with different encodings but >> no success yet. I guess I first have to understand _exactly_ what is >> going on before I can solve it. > > To review how this is supposed to work, the language examples (Tcl in > this case) should encode each PLplot user string as a UTF-8 string, > and that string should be passed directly without transformation to > the PLplot C level as a UTF-8 string by the bindings (Tcl in this > case). The PLplot C core library then translates the UTF-8 string > into an unsigned 32-bit (PLUNICODE) array consisting of FCI's > (representing the ascii font-changing commands embedded in the string) > and UCS4 indices (representing the UTF-8 unicode encoding in the > string). Our modern device drivers such as cairo and qt then interpret > that array further to make the appropriate library call for an FCI to > change the font or to use the UCS4 index to access a particular glyph > for TrueType fonts. There is one more layer of indirection for -dev > psc; it translates the UCS4 indices into indices that are suitable for > indexing Type 1 fonts with the overwhelming majority of results being > the index for Type 1 blank because Type 1 fonts have an extremely > limited number of glyphs they can represent. So if there was some > screwup in encoding at the Tcl level that generates random UCS4 > indices, the most probable response for -dev psc would be a blank > glyph which would be rather useless for diagnostic purposes. So I > suggest it would be much better to use one of the cairo or qt devices > to diagnose what is going wrong at the Tcl example 24 or bindings > level since the result of random UCS4 garbage would be more likely to > end up as non-blank glyphs. > I solved it, based on a suggestion from a fellow Tcler: we need to pass the string to a Tcl_UtfToExternal*() function. The problem on Windows is that the system encoding is _not_ UTF-8, but cp1252. The strings in the source are therefore translated from cp1252 to UTF-8, so that the resulting string represents these characters. Using said function undoes this translation: the PLplot routines receive the original sequence of bytes. I have not implemented it yet in pltclgen.tcl, but that should be easy enough. (I have instead tested it in the _generated_ source and that gives results that are identical to the C output.) Could someone check that this is the case on Linux as well? Here is the adjusted source for the plptex wrapper (use this instead of the generated source in tclgen.c): /*--------------------------------------------------------------------------*\ * plptexCmd * * Processes plptex Tcl command. \*--------------------------------------------------------------------------*/ static int plptexCmd( ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[] ) { PLFLT x; PLFLT y; PLFLT dx; PLFLT dy; PLFLT just; const char *text; int i; Tcl_DString dstring; pl_errcode = 0; errmsg[0] = '\0'; if ( (argc == 2) && (strlen(argv[1])>0) && (strncmp(argv[1],"-help",strlen(argv[1])) == 0) ) { Tcl_AppendResult( interp, "command syntax: \"", "plptex x y dx dy just text", "\"", (char *) NULL); return TCL_ERROR; } if ( (!0 && 0 && (argc < (1 + 6 - 0))) || (!0 && !0 && (argc != (6 + 1))) || ( 0 && (argc != 1) && (argc != (6 + 1))) ) { Tcl_AppendResult( interp, "wrong # args: should be \"", "plptex x y dx dy just text", "\"", (char *) NULL); return TCL_ERROR; } x = atof(argv[1+0]); y = atof(argv[1+1]); dx = atof(argv[1+2]); dy = atof(argv[1+3]); just = atof(argv[1+4]); Tcl_DStringInit( &dstring ); Tcl_UtfToExternalDString( NULL, argv[1+5], (-1), &dstring ); text = Tcl_DStringValue( &dstring ); plptex ( x, y, dx, dy, just, text ); Tcl_DStringFree( &dstring ); if (pl_errcode != 0) { Tcl_AppendResult(interp, errmsg, (char *) NULL); return TCL_ERROR; } plflush(); return TCL_OK; } If this works on Linux too, then I am going to adjust it in the generator script (otherwise I would have to add an #ifdef/#endif). Another question: We do support unicode/UTF-8 strings in all other cases where an argument is a string, right? Like: plbox taking two labels. Then I suggest that I change all occurrences where a non-ASCII might arise (it seems superfluous for the x/y options to plbox). >> >> I do not see differences with examples 17 and 29 on plain Windows, >> though. > > Is this a 32-bit or 64-bit result? > 32-bits. >> >>> >>> In sum, probably example 17 and 24 need some more work on Tcl for >>> 32-bit Windows platforms (and possibly 32-bit Linux platforms) while a >>> change of all examples 29 to shift the discontinuties slightly from >>> the exact centre of the plot will probably solve that (32-bit Tcl?) >>> issue. Aside from these rendering issues it looks like everything >>> works at least on my wine version of the Windows platform, and that >>> good result needs confirmation on a Microsoft version of the Windows >>> platform as well. >> >> Example 17's differences are a bit puzzling. I have no clue yet. >> I can reproduce differences with example 29, so that is easier. > > Can you clarify that last sentence which is in apparent contradiction > with what you said about no differences for example 29 above? > Oops, I meant example 26. Regards, Arjen DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |
From: Alan W. I. <ir...@be...> - 2010-12-21 09:55:23
|
On 2010-12-21 09:25+0100 Arjen Markus wrote: > I solved it, based on a suggestion from a fellow Tcler: we need to > pass the string to a Tcl_UtfToExternal*() function. The problem on > Windows is that the system encoding is _not_ UTF-8, but cp1252. > The strings in the source are therefore translated from cp1252 to UTF-8, > so that the resulting string represents these characters. Using said > function undoes this translation: the PLplot routines receive the > original sequence of bytes. I think this explanation must be incomplete or incorrect. The reason I say this is our Tcl example files, e.g., x24.tcl, (and our C code as well, for that matter) are all encoded directly in UTF-8. So when I use the emacs editor to look at x24.tcl, the "Peace" words below set peace { are displayed correctly for every language. Apparently it would be impossible to encode those peace words as cp1252. That encoding is quite limited and very close to ISO-8559-1 according to the Wikipedia article about it, and thus would not be able to represent, e.g., the Mandarin, Arabic, etc. words for Peace that I see when editing that file with emacs. On Linux, when the pltcl executable interprets those UTF-8 encoded source files, it does everything as expected and passes those strings directly as UTF-8 to the PLplot core library. (I have just double-checked that by running "../../utils/pltcl x24 -dev pngcairo -o test.png" in the build-tree examples/tcl subdirectory on Linux, and the result was a perfect Peace flag just like we get on Linux with the x24c executable that is written in C (as can be seen by looking at, e.g., http://plplot.sourceforge.net/examples-data/demo24/x24.01.png). Arjen, what happens for the x24c executable on Windows for a modern device such as pngcairo? (My wine fonts are quite limited so I just get blanks for the exotic languages so it is not a definitive test). If your x24c results with pngcairo are similar to the above result displayed on our web site, then all is well in Windows C, and the only remaining question is what is going on in Tcl for Windows that is so different from C for Windows in how it deals with source files that are encoded in UTF-8. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
From: Arjen M. <arj...@de...> - 2010-12-28 10:10:37
|
Hi Alan, Tcl has two options to do what you suggest: - Set the system encoding to cp1252 - Source the file using -encoding cp1252 (A third is to adapt the C side of the Plplot API.) I will experiment with both. Regards, Arjen On 2010-12-24 22:20, Alan W. Irwin wrote: > On 2010-12-24 13:05+0100 Arjen Markus wrote: > >> Hi Alan, >> >> On 2010-12-21 20:03, Alan W. Irwin wrote: >> >>> >>> I will stop now and not comment more on the Tcl case, because I think >>> it is essential to focus on C for now and one of the cairo or qt >>> devices. >>> >>> Of course, as I have stated before the psc device driver is not useful >>> for diagnosis of encoding issues because everything exotic such as the >>> Mandarin Peace word ends up as blanks in any case because the standard >>> Type 1 fonts that the psc device uses have an extremely limited glyph >>> set that does not include Mandarin glyphs or any other non-English >>> glyphs besides Greek (for mathematical purposes). >>> >> >> I have checked the C and F95 examples 24 with the wxWidgets device on >> Windows: they work very nicely, except that my PC does not have all the >> required fonts. With Tcl I recognise the odd sequences I also see in >> the source code (viewed from a cp1252 perspective). > > Thus, it appears from your results that C and Fortran on Windows > simply accept the byte sequences mentioned in strings without > molesting them while your hypothesis is that Tcl does not follow that > simple model. Instead it does an implicit transformation of all > strings from (presumed) system encoding to UTF-8 which messes up the > byte sequences, and your proposed cure is to take all strings input to > PLplot from Tcl and do the inverse transformation which uses > a call to Tcl_UtfToExternalDString with a > NULL encoding. From the man page, what that will do is to convert a > (presumed) UTF-8 string to the (presumed) system encoding. > > If your hypothesis is correct, then your proposed cure might indeed > work on all platforms. Certainly on Linux, the implicit > transformation is UTF-8 to UTF-8 or the identity transform (which is > why Tcl works right now for example 24 on Linux), and your inverse > transformation would also be the identify transformation on Linux and > should therefore also work on that platform. > > However, I am concerned with the following issues. > > 1. All PLplot API arguments that are strings are assumed to be in > UTF-8. Thus, the call to Tcl_UtfToExternalDString with NULL has to be > made in the Tcl bindings for _every_ function in the PLplot API that > has an input string. > > 2. Does the implicit transformation work for arbitrary UTF-8 (e.g., > arbitrary series of 8-bit bytes) or are there some 8-bit bytes which > cannot be validly interpreted as cp1252 or which have special > meanings. > > 3. Is Tcl_UtfToExternalDString with NULL encoding the exact inverse > of the implicit transformation? > > All of these issues can be dealt with. Obviously some care in the Tcl > bindings should take care of issue 1 and to alleviate concern about > issues 2 and 3 completely, it would be a good idea to put together a > complete test of all 256 possible 8-bit character combinations. I am > thinking along the lines of generating a file from C with a string of > all possibilities from 255 down to a zero (to terminate the string). > Then using an editor copy that exact string of 256 bytes to Tcl source > code that automatically puts that string through the implicit > transformation. Then use the Tcl_UtfToExternalDString with NULL to > transform that string before calling a C programme that simply outputs > the string. Then compare that output file with the original file with > 256 characters to see if you get all 256 characters back in their > original form. > > However, to avoid this work it would be better to convince Tcl not to > do the implicit string transformation in the first place. The way > this is handled in Python is to put the following string in the > first or second line of the Python script that identifies the > whole Python source file is encoded in utf-8: > > # -*- coding: utf-8 -*- > > We do that for the following Python examples: > > software@raven> grep coding: examples/python/xw??.py > examples/python/xw18.py:# -*- coding: utf-8; -*- > examples/python/xw24.py:# -*- coding: utf-8; -*- > examples/python/xw26.py:# -*- coding: utf-8; -*- > examples/python/xw33.py:# -*- coding: utf-8; -*- > > I think Tcl may have something equivalent in the > > encoding system utf-8 > > command. From the documentation users are discouraged from using that > command because it affects everything such as system calls. For > example, puts would output strings in UTF-8 encoding rather than the > actual (e.g., cp1252 on your platform) system encoding on Windows > machines. But is that actually an issue for the above examples? > First, we don't interact with the operating system (e.g, with puts) as > far as I know with those examples, and UTF-8 and cp1252 coincide in > any case for ascii strings. Anyhow, if "encoding system utf-8" works > for those examples, I think we should use it rather than the more > difficult steps outlined above. Of course, we should inform Tcl users > browsing our example code via a comment in those examples that PLplot > requires utf-8 system encoding for all non-ascii input strings. > >> >> Over the weekend I won't be able to do anything - holiday >> obligations :). >> >> I wish you all a merry Christmas and a happy New Year. > > I wish everybody here a "Merry Christmas" and "Happy New Year" as well. > > Enjoy your holidays, Arjen, and I hope when you come back you will > find the "encoding system utf-8" solution works without issues for the > affected examples. > > Alan > __________________________ > Alan W. Irwin > > Astronomical research affiliation with Department of Physics and Astronomy, > University of Victoria (astrowww.phys.uvic.ca). > > Programming affiliations with the FreeEOS equation-of-state implementation > for stellar interiors (freeeos.sf.net); PLplot scientific plotting software > package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of > Linux Links project (loll.sf.net); and the Linux Brochure Project > (lbproject.sf.net). > __________________________ > > Linux-powered Science > __________________________ > DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail. |