#2614 pdflatex internal error when doing an \includegraphics of some tricky filename

[Vincent Belaïche]

I am on MSWindows 10. Steps to reproduce the bug :

1. Open attached miktex-bug.zip
2. Extract filename.zip and install the package — this is an experimental package not yet on CTAN.
3. Extract the 4 other files
4. Run MSDos batch file doit.bat, it will create a number of directories and compile two files, one is works.tex which compile fine, and the other is doesnt_work.tex which causes a pdflatex internal error.

I need to create all these funny directories to reproduce the bug because even with quite funny directory names it has always worked fine. It seems that there is a combination of path length and use of UTF-8 in filenames which causes the bug.

My guess : probably at some point memory is allocated to store the filename with a confusion between character count and byte count, so the memory is wrongly allocated, meaning not big enough, and this causes some crash.

[Christian Schenk]

Your example is too complex, i.e., I am not able to reproduce the bug.
You should provide a minimal example and a trace (DebugView, log files, screen shots).

And please note: Latin-1 characters in a TeX input file are not portable (in case you want to
upload these input files to CTAN).

[Vincent Belaïche]

Please find attached the session log.

The pdflatex.log file contains the following statement on the last line :

2017-07-18 17:12:32,954+0200 FATAL pdflatex - MiKTeX encountered an internal error.

I cannot understand your statement that "Latin-1 characters in a TeX input file are not portable", do you mean "file name", or "file" ?

[Christian Schenk]

Still cannot reproduce the bug with the given info.

Please attach the files which were written to the MiKTeX log directory:
%LOCALAPPDATA%\miktex\2.9\miktex\log

[Vincent Belaïche]

Here is the whole MikTeX log directory zipped.

[Vincent Belaïche]

One more thing, the attached zip is the zipped directory Vincent under c:\BugMT.
So if you unzip it under c:\BugMT, you will find the compilation logs in directory c:/BugMT/Vincent/AppData/Local/Temp/efrench/bug-miktex/abcdefghij/abc_defghij/abcdef/abcdefghij/2016_Abcdef_Géhijk/ as if you had run the test on your machine.

[Christian Schenk]

I still have problems to understand your test case. Running

pdflatex doesnntwork.tex

yields

an LaTeX error:

! LaTeX Error: File 'filename.sty' not found.

I cannot fix this. filename.sty cannot be found in the MiKteX distro, I give up.

[Vincent Belaïche]

You get this error message because you have not followed the steps to reproduce the bug supplied in with my initial message when creating this report:

1. Open attached miktex-bug.zip
2. Extract filename.zip and install the package — this is an experimental package not yet on CTAN

I attach again filename.zip. If you do not want to install it, just place the content into the c:/BugMT/Vincent/AppData/Local/Temp/efrench/bug-miktex/abcdefghij/abc_defghij/abcdef/abcdefghij/2016_Abcdef_Géhijk/20170711_aébcdefghijklm_nopgrst directory and compile doesnt_work.tex.

[Vincent Belaïche]

Hello,

I have made it even simpler for you. I have put all the filename.sty package files in the attached miktex-bug.zip, and I have modified the doit.bat script so that they are copied to the c:/BugMT/Vincent/AppData/Local/Temp/efrench/bug-miktex/abcdefghij/abc_defghij/abcdef/abcdefghij/2016_Abcdef_Géhijk/20170711_aébcdefghijklm_nopgrst directory along with the doesnt_work.tex and works.tex files before calling pdflatex.

I have also modified doit.bat so that it asks whether or not removing any exisring c:\BugMT directory before starting the test, so as to run it on a clean state. But if you don't agree with removal, that is not an issue, the script will just bark a little at making existing directories.

So, to reproduce the problem, the only things that you have to do is:

1. Unzip the attached miktex-bug.zip
2. Open an MSDos console in the unzipped miktex-bug directory, and call doit.bat from there.

[U_Fischer]

Your script doesn't work for me. The folders are created but then the copy instructions fail as the system "can't find the path" and the folder are empty.

(Sorry click too fast)

I copied the files manually and also tried to create your abûc_défég.pdf with a shorten version of your batch file. Neither works nor doesn-works.tex compile: both can't find their graphic, but I get no crash.

[Vincent Belaïche]

Hello Ulrike,
Glad to here from you. Are you on MSWindows 10 ?
What is your character code page (open a console and type chcp to know it).
I have attached another version where the script is slightly modified to set character code page explicitely to 1252, so that to be consistent with the naming of the directories used in the script.

[U_Fischer]

I'm on windows 10. My codepage is 850 but changing the codepage to 1252 or unicode doesn't change the behaviour: the script still doesn't copy the files. Be aware that my hard disk is nfss formatted, so my file names are (probably) unicode.

Beside this: This discussion is not new to me. For years people have been claiming that it should be possible to use file names with non-ascii chars without fuss. There is a chance to get it working in the future when/if everything uses utf8 -- then one can use \detokenize --, but you seem to want to get it working with latin encoded scripts (and perhaps file names) too and this is imho bound to fail. You simply don't know what happens on other computers and how it stores such files and folder names - the failure of your script shows this quite clearly.

[Vincent Belaïche]

Filenames are also Unicode encoded on my machine. MSDos is supposed to make the conversion from the current MSDos code page to the file-sytem encoding. And the filename.sty package is also doing the conversion from latin9 to unicode. So everything should go to unicode when getting to the file-system, and this is why the work.tex file compiles fine.

About your comment, the problem is not whether what I do is silly, crazy, or bound to fail, the problem is that pdflatex is crashing, and a crash is a bug whatever you have done to make it crash. I am not asking that people are doing this or that when they name their files, just for my own purpose I find it practical to have filenames not in plain ASCII, and I also find it practical to use latin9 which is plainly sufficient to write French.

Maybe your problem with the doit.bat is that as a user you don't have the access right to create a C:\BugMT directory. What happens when in a console you type manually : mkdir C:\BugMT?

If it fails, then maybe you should replace C: by D: or some other drive where you can create a sub-dir.

I have spent quite a lot of time to build this minimal example — OK you may find it not so minimal — from a document with plainty of text and graphics inclusions. It boiled down that the problem is not the graphic itself but the file-name.

[U_Fischer]

As I wrote: the folders are created. And I can copy files in this folders. So there is not an access problem. But I found the reason: I was starting the script from another partition. After I moved it to C: the files where copied. But still both tex files don't find the graphics and neither crash.

Beside this: I believe you that you get a crash. Over the years I have seen a lot of

   FATAL pdflatex - Conversion from UTF-8 byte sequence to wide character string did not succeed.

(which would be my guess that you have this in you log in AppData/Local/MiKTeX/2.9/miktex/log/pdflatex.log too) with much simpler setups as yours when a graphic with non-ascii chars was included and I simply don't want to spent more time on it.

[Vincent Belaïche]

@U_Fischer

Dear Ulrike,

First of all, thank you for answering, and for finding the reason why the doit.bat script did not succeed in copying the files.

Second, your statement « I believe you that you get a crash » is a misunderstanding of what I am undergoing. Let us not call a crash that sort of situation which you are describing. Rather we should consider this not really a crash but as an error message which pdflatex produces when the users give a filename that is not properly encoded. So, even though it is a brutal exit looking like a crash, it is an exit triggered by design on a fatal situation with some reason that pdflatex has detected.

The situation which I am describing is of a different nature, as it happens without any understandable cause.

The filename.sty experimental package which I am developping is a transcoder from some input encoding (here latin9) to the file-system encoding (here UTF-8). So, when debugging the filename.sty package I often had the error message which you are mentioning:

FATAL pdflatex - Conversion from UTF-8 byte sequence to wide character string did not succeed.

To me, this error clearly means a bug in filename.sty not transcoding properly, not a problem with pdflatex.

So, please, believe me: I am not bothering you with a bug of mine. This « conversion did not succeed » is not the error message that I get in %LOCALAPPDATA%\miktex\2.9\miktex\log\pdflatex.log. After deleting the whole content of the directory %LOCALAPPDATA%\miktex\2.9\miktex\log, there is only one file created after the crash, this file is pdflatex.log, and it contains a single line as follows:

2017-07-19 12:18:36,599+0200 FATAL pdflatex - MiKTeX encountered an internal error.

Maybe the bug is dependant of the pdflatex version and this is why you can't reproduce it, pdflatex --version gives this on my machine:

MiKTeX-pdfTeX 2.9.6100 (1.40.17) (MiKTeX 2.9.6100 64-bit)
Copyright (C) 1982 D. E. Knuth, (C) 1996-2016 Han The Thanh
TeX is a trademark of the American Mathematical Society.
compiled with zlib version 1.2.8; using 1.2.8
compiled with libpng version 1.6.24; using 1.6.24
compiled with poppler version 0.46.0
compiled with jpeg version 8.4

Maybe I should try to install the latest MikTeX and see if I can reproduce the crash.

I am surprised that your compiling works.tex and doesnt_work.tex fails on both without a crash but with not finding the graphics. Could you please send to me the log files, so I can see what message filename.sty outputs.

[U_Fischer]

Sigh. Are you saying that I just spent the morning trying to reproduce a problem that you didn't tested on a current miktex? What sense has it to make a bug report for an outdated miktex?

[Vincent Belaïche]

:-$ Oooops… time goes by so fast, I was not really conscious that my MikTeX was so old.
I feel so sorry to have you loose your time, I hope that you can forgive me for that.

I must admit that I had completely overlooked that point, once I had got convinced that it was a crash I had spent all my efforts trying to build a minimal example.
I am currently downloading the latest MikTeX, and I let you know about the bug as soon as I get it installed.

[Vincent Belaïche]

OK, I have re-installed the latest MikTeX from scratch — even later than latest installer, as I ran the update manager right after installation. And the good news is that both works.tex and doesnt_work.tex compile fine without any issue.

So, hopefully the bug has been corrected during the almost past year (my previous installation was not so outdated, it was of 2016-11-15), or maybe it is still there and has just gone silent because the memory is handled slightly in a different way (not the same order of mallocs may result in no crash…).

Who knows…

I will try to make some script that generates random path in order to make a torture test.

@Christian Schenk: have made any fix since 2016-11-15 which may explain the bug disapearance ?

[Christian Schenk]

Yes, there have been some bug fixes regarding special character in file names.

[Vincent Belaïche]

OK, if it sounds likely that these bug fixes have removed the possbility of a dirty crash when some filename has special characters, then it is probably useless if I spend time trying to make some torture test to reproduce the problem with the latest version. By dirty crash I means that sort of crash that makes the « 2017-07-19 12:18:36,599+0200 FATAL pdflatex - MiKTeX encountered an internal error. » sort of error message.

Please let me know if you have any doubt and want me to do that yet.

Sorry for the disturbance, and thank you so much to both for your time.

[Vincent Belaïche]

@U_Fischer
There remains one unclear point however. How comes that Ulrike does not get the same result as me.
Ulrike : do you have the latest version of pdflatex ;-D ? Mine is now:

MiKTeX-pdfTeX 2.9.6362 (1.40.18) (MiKTeX 2.9.6350 64-bit)
Copyright (C) 1982 D. E. Knuth, (C) 1996-2016 Han The Thanh
TeX is a trademark of the American Mathematical Society.
using bzip2 version 1.0.6, 6-Sept-2010
compiled with curl version 7.54.0; using libcurl/7.54.0 WinSSL
compiled with expat version 2.2; using expat_2.2.0
compiled with jpeg version 9.2
compiled with liblzma version 50020032; using 50020032
compiled with libpng version 1.6.29; using 1.6.29
compiled with libressl version LibreSSL 2.5.3; using LibreSSL 2.5.3
compiled with MiKTeX Application Framework version 1.6334; using 1.6334
compiled with MiKTeX Core version 1.6344; using 1.6344
compiled with MiKTeX Archive Extractor version 1.6300; using 1.6300
compiled with MiKTeX Package Manager version 1.6353; using 1.6353
compiled with poppler version 0.55.0
compiled with uriparser version 0.8.4
compiled with zlib version 1.2.11; using 1.2.11

Another possibility is that you have some version of MSDOS that works differently than mine w.r.t. to non ASCII characters — I heard that some Asian PCs do something like that, but I don't know really the difference. I have attached an MSDOS script to do the test, if will ask you to delete & recreate a subdirectory named test and will create into this subdirectory two empty files, one named fic€ and the other named ficÇ.

The script file is encoded in windows-1252 and basically does that:

chcp 1252
type NUL > fic€
chcp 850
type NUL > fic€

It uses the same character code 128 (aka 200'o, aka 80'h) for the last character in the fic∗ filename, but this does not create the same character because the first time code page is 1252, so the € caracter is taken, and the second time code page is 850, so the Ç is taken. This is like Shakespear and Cervantes dying at the same date, but not on the same day because Spain had adopted the Gregorian calendar earlier than England.

If you have the latest pdflatex, and by running this test.bat script you get the two files, then we have an unresolved problem. If by running the test.bat script you do not get the two files, then your problem might be due to your MSDOS version doing something-else than mine — anyway, using MSDOS was a very bad idea. If so, the LaTeX compilation issue is that the doit.bat script does not create the graphics in the place where works.tex and doesnt_work.tex expect them.

[U_Fischer]

You shouldn't misuse the miktex bug tracker for such discussions.

[Vincent Belaïche]

You wrote:

I copied the files manually and also tried to create your abûc_défég.pdf with a shorten version of your batch file. Neither works nor doesn-works.tex compile: both can't find their graphic, but I get no crash.

• Your files get a compilation failure with no crash due to graphics not found
• My files compile fine.

There is a discrepancy between you and me, and as long as we don't know the cause, whether it is MikTeX or something else, we cannot say that seeking the real cause is a misuse of the MikTeX bug tracker.

But, if you just do not want to spend any more time at it, I fully respect this, provided that you state it clearly w/o unfairly blaming me with misusing the bug-tracker.
BTW, I do not want either to continue this discussion :-(.