#240 Version 3.0.3 encoding test failing: "Source not found"

[Giovanni Mariani]

Trying to build the program for Rosa Fresh R9 (aka Rosa 2016.1) on x86_64.
After resolving issue #239, the build still fails at the testing phase, at the test-svcore-data-fileio stage (attached a full log of the failure).

There are two types of failures: in EncodintTest and in MIDIFileReaderTest.
It seems that the first one is unable to find some test files; eg:
FAIL! : EncodingTest::readAudio(T??mple of Sp??rks.mp3) 'reader != nullptr' returned FALSE. () Loc: [o/../svcore/data/fileio/test/EncodingTest.h(109)] FAIL! : EncodingTest::readAudio(T??mple of Sp??rks.ogg) 'reader != nullptr' returned FALSE. () Loc: [o/../svcore/data/fileio/test/EncodingTest.h(109)] AudioFileReaderFactory::createReader("svcore/data/fileio/test/encodings/T??mple of Sp??rks.mp3": Source not found AudioFileReaderFactory::createReader("svcore/data/fileio/test/encodings/T??mple of Sp??rks.ogg": Source not found AudioFileReaderFactory::createReader("svcore/data/fileio/test/encodings/?????????????????????.mp3": Source not found AudioFileReaderFactory::createReader("svcore/data/fileio/test/encodings/?????????????????????.ogg": Source not found FAIL! : EncodingTest::readAudio(?????????????????????.mp3) 'reader != nullptr' returned FALSE. () Loc: [o/../svcore/data/fileio/test/EncodingTest.h(109)] FAIL! : EncodingTest::readAudio(?????????????????????.ogg) 'reader != nullptr' returned FALSE. () Loc: [o/../svcore/data/fileio/test/EncodingTest.h(109)]
The same, but without the message error "Source not found" looks like for MIDIFileReaderTest:

FAIL!  : MIDIFileReaderTest::read(?????????????????????.mid) 'm != nullptr' returned FALSE. ()
   Loc: [o/../svcore/data/fileio/test/MIDIFileReaderTest.h(81)]

I would also add that all the files are present and all have a filename containing international chars... UTF support issue?

BTW, is there a way to optionally disable the test execution at build time? So could continue to package sonic-visualizer for Rosa...

[Luigi Baldoni]

I have the same problem on openSUSE. See attached patch to disable tests.

[Giovanni Mariani]

Thank you for the workaround...

However this does not solve the issue, that in turn is twofold:
a. The test failing to find/open test files that are present;
b. There is no way to avoid running the test, by passing an opportune switch to the build process.
The first one is the real thing, while the latter would improve the actual situation wtihout the need to patch away all the test...

The ball is still in the devs camp, I think.

[Chris Cannam]

Hm. What is Rosa Fresh? Does it do something unusual (among contemporary systems) such as using a non-UTF8 encoding for the filesystem?

(Edit: Oh, I missed the reference to OpenSUSE in a reply. Well, it would also be good to check what filesystem encoding is in use there. These tests probably will not work at all if the filesystem is not UTF8 - I'll have to do something to work around that, but it would be good to know first whether that is the cause of the problem or not.)

[Luigi Baldoni]

Any recommendation on how to check that?

[Giovanni Mariani]

Rosa Fresh is a distribution derived from the good old Mandriva one, started at the same time OpenMandriva; see: http://wiki.rosalab.ru/en/index.php/Main_Page

About the questions you made below, I will attach here a little logfile with the results of the tests you suggested.
My system installation has as language "enghlish US", but I have an italian keyboard and wanted the european date, numbers, and so o on...and this in turn should explain the only difference between my logfile and the setting of Rosa build VM: I have "LANG=it_IT.UTF-8", where they have "LANG=en_US.utf8".

ATM I have worked around the issue at hand by using the patch from Luigi Baldoni to disable the tests...

[Chris Cannam]

One way might be to look at what the filenames actually consist of in stream-of-byte terms:

$ ls svcore/data/fileio/test/encodings/ | od -x

(Also, as a sanity check - if you just run "ls" in that directory, do the filenames appear with non-ASCII characters correctly or are they already mangled?)

And check the output of

$ locale

which may or may not include an expected filesystem encoding (the part after the dot in the locale string, e.g. "en_GB.UTF-8").

Perhaps if you could paste the output of both of those?

[Luigi Baldoni]

Output of the commands above in the same build environment in the attached log.

[Chris Cannam]

Thanks. So the filenames are UTF-8 encoded but the locale is C, which I think (?) may mean ASCII filenames are expected. I guess that Qt is translating to ASCII (and losing information) rather than UTF-8 when referring to filenames, then... assuming it is using the locale encoding for filenames as well as for content encoding. Either way, that should give me something to start from.

Running with only C locale rather than a country locale seems like an unusual choice - are you doing so for some particular reason?

What happens if you do something like

$ export LC_ALL=en_US.UTF-8

before running the build?

[Luigi Baldoni]

Thanks, that did the trick.

With regard to your question, that's the default environment in the VM where openSUSE builds its packages.
Perhaps Rosa uses something similar?