TestLily / Backup-2016-12-10 / #4751 Import Philomelos enhancements to musicxml2ly

Werner LEMBERG - 2016-01-24

Ahem, where's the patch?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Simon Albrecht - 2016-01-24

status: New --> Accepted
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Simon Albrecht - 2016-01-24

The patch is under construction. We can still have issues without having a patch :-)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Gourlay - 2016-05-03

Patches are now ready for review:
http://codereview.appspot.com/295120043

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Werner LEMBERG - 2016-05-03

Thanks for the patch, but this is very hard to review to its enormous size. I assume that you got a series of commits, right? Perhaps it is possible to add this directly to lilypond's git repository as a branch (e.g. into 'origin/dev/johngourlay/issue-4751').

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Gourlay - 2016-05-03

I tried to create the branch you requested in the lilypond repository but it didn’t work. I haven’t tried this kind of thing before, so I might be doing something wrong. Or, maybe I don’t have the rights needed to push anything to the repository. Do you have any suggestions? Following is my git command and the output it produced:

[lilypond-git (master %)]$ git push origin master:dev/johngourlay/issue-4751
Counting objects: 100, done.
Compressing objects: 100% (82/82), done.
Writing objects: 100% (100/100), 76.20 KiB | 0 bytes/s, done.
Total 100 (delta 77), reused 23 (delta 18)
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Werner LEMBERG - 2016-05-03

Yes, you don't have write access. Either you create a public clone somewhere so that you can publish your branch, or you ask the lilypond-devel list for write permission.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Gourlay - 2016-05-06

The branch dev/johngourlay/issue-4751 should now exist.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thomas Morley - 2016-05-07

status: Accepted --> Started

Patch: --> new
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-05-07

This passes make but not make check. I cannot fathom why it fails but maybe if you run the test yourself you can work it out.

See the section: http://lilypond.org/doc/v2.19/Documentation/contributor-big-page.html#compiling-regression-tests

You don't need to make doc (I'll test that for you when you submit the patch again).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-05-07

Patch: new --> needs_work
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Gourlay - 2016-05-07

For me “make check" also fails. What’s the difference between “make check” and “make test”? The latter works, as does “make doc”. I don’t see any documentation for “make check”.

The tail of the output I get from “make check” looks like this:

Converting MusicXML file `/home/john/lilypond-git/input/regression/musicxml/99b-Lyrics-BeamsMelismata-IgnoreBeams.xml'...

Writing snippets...
Processing...
Processing /home/john/lilypond-git/build/out/lybook-testdb/snippet-names-1619550309.ly
command failed: /home/john/lilypond-git/build/out/bin/lilypond -I /home/john/lilypond-git/input/regression/musicxml -dbackend=eps --formats=ps -dseparate-log-files -dinclude-eps-fonts -dgs-load-lily-fonts --header=texidoc -I /home/john/lilypond-git/Documentation/included/ -ddump-profile -dcheck-internal-types -ddump-signatures -danti-alias-factor=1 -I "./" -I "/home/john/lilypond-git/build/input/regression/musicxml" -I "/home/john/lilypond-git/input/regression/musicxml" --formats=eps -deps-box-padding=3.000000 -dread-file-list -dno-strip-output-dir "/home/john/lilypond-git/build/out/lybook-testdb/snippet-names-1619550309.ly"
Child returned 1
Error ignored by lilylib
Error trapped by lilypond-book

Please see /home/john/lilypond-git/build/out/lybook-testdb/snippet-names-1619550309.log

/home/john/lilypond-git/./make/ly-rules.make:30: recipe for target 'out-test/collated-files.texi' failed
make[2]: [out-test/collated-files.texi] Error 1
make[2]: Leaving directory '/home/john/lilypond-git/build/input/regression/musicxml'
/home/john/lilypond-git/./make/lysdoc-targets.make:12: recipe for target 'local-test' failed
make[1]: [local-test] Error 2
make[1]: Leaving directory '/home/john/lilypond-git/build/input/regression/musicxml'
/home/john/lilypond-git/GNUmakefile.in:306: recipe for target 'test' failed
make: *** [test] Error 2

The log file which I’m supposed to see contains nothing except the LilyPond banner, “GNU LilyPond 2.19.41”.

On May 7, 2016, at 7:40 AM, pkx166h lilypond-pkx@users.sf.net wrote:

This passes make but not make check. I cannot fathom why it fails but maybe if you run the test yourself you can work it out.

See the section: http://lilypond.org/doc/v2.19/Documentation/contributor-big-page.html#compiling-regression-tests

You don't need to make doc (I'll test that for you when you submit the patch again).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thomas Morley - 2016-05-07

As far as I understand:

make test
compiles the regression-tests, checking for errors. According to
http://lilypond.org/doc/v2.19/Documentation/contributor/compiling-regression-tests

To check whether your changes are all ok or whether you introduced some bads you should compare the regression-tests compiled with and without your changes:

make test-baseline
done in unchanged master will establish the baseline to check against.

make check
done after all your changes applied, will compile the regtests again, compare them with their counterpart from the baseline and return a file with the details. According to
http://lilypond.org/doc/v2.19/Documentation/contributor/regtest-comparison

Maybe of some help:
While running make check the first displayed lines read:

For tracking crashes: use grep sourcefilename `grep -L systems.texi out/lybook-testdb/*/*log|sed s/log/ly/g`

Although I was told it may be overly complicated, doing so sometimes served me well.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-05-07

John,

You have to do make, make test-baseline, then apply your patch, then do another make and them make check.

If you just do make, make test then make check I am not sure what happens.

It depends on where the error happens as to how easy it is to decipher.

Somtimes the name of the file that fails the make check is the clue (i.e. if you tried to just compile the file with your patch applied it would fail) but tracking which file is the culprit can be a bit of an art.

James

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Gourlay - 2016-05-08

Thomas and James,

Thanks for the Contributor’s Guide reference. So much for the search feature. I did a search for make check and got nothing obviously useful.

With the explanation that make check compares the before and after output of LilyPond on the regression tests, it’s obvious to me what’s causing the error. The new version of musicsml2ly supports many features of MusicXML that were ignored by the old version. The ly output of the new musicxml2ly on some of the xml files (I don’t know which ones) will certainly be different from the output of the old musicxml2ly. How do we overcome this hyper vigilance on the part of make check?

John

On May 7, 2016, at 5:22 PM, pkx166h lilypond-pkx@users.sf.net wrote:

John,

You have to do make, make test-baseline, then apply your patch, then do another make and them make check.

If you just do make, make test then make check I am not sure what happens.

It depends on where the error happens as to how easy it is to decipher.

Somtimes the name of the file that fails the make check is the clue (i.e. if you tried to just compile the file with your patch applied it would fail) but tracking which file is the culprit can be a bit of an art.

James

[issues:#4751] Import Philomelos enhancements to musicxml2ly

Status: Started
Created: Sun Jan 24, 2016 02:27 AM UTC by John Gourlay
Last Updated: Sat May 07, 2016 11:40 AM UTC
Owner: John Gourlay

Import Philomelos enhancements to musicxml2ly from github.com/Philomelos/lilypond-musicxml2ly-dev. This includes the following:
- New command line option --transpose c TOPITCH.
- Added the sound tempo recognition for midi output.
- Added support for standalone sound elements.
- Added the --shift-meter option to make the music look faster/slower.
- Implemented recognition of stem values "up" and "down".
- Added support for ChordNames transposition.
- Added the command line options --tc / --tabclef [tab|moderntab]
to be able to switch between the two styles of the tab clef.
- Added transpose support for FretDiagrams.
- Allow only "moderntab" and "tab" for tab_clef value.
- Added the command line options --sn / --string-numbers [t|f] to activate
(default) | deactivate the string number stencil.
- Added conversion of elements to a separate FretBoards voice.
- No longer use the staff tuning option.
- Added --no-stem-direction option to ignore stem directions from MusicXML.
- Added <credit> elements recognition.
- Added page layout handling options.
- Fixed the issue with spaces/brackets in filename.
- Recognize the print-lyric attribute.
- Colored noteheads.
- Allow both <fermata>angled</fermata> and <fermata type="angled"></fermata>
and convert them correctly.
- Color-attribute ignored. Color of notehead and stem can be set with
hexadecimal strings. Color of stem is set even if
conversion_settings.convert_stem_direction is False.
- Include articulate.ly if options.midi == True.
- If a <instrument-sound>-tag is present, attempt to map its value to a
corresponding Lilypond midi-instrument and assign it to the correct staff.
- Automatically include the philomelos tagline in the header.
- Fix numerous bugs.</instrument-sound></credit>

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/testlilyissues/issues/4751/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John,
although your patch is beyond my depth I tried to do some checking. Thus I checked out your branch. But it failed make.
I then reran sh autogen.sh --noconfigure on it and ../configure in \build.
Though it still fails:

[...]
echo /home/hermann/lilypond-git/build/scripts/build/out/help2man
/home/hermann/lilypond-git/build/scripts/build/out/help2man
/home/hermann/lilypond-git/build/scripts/build/out/help2man out/musicxml2ly > out/musicxml2ly.1
help2man: can't get `--help' info from out/musicxml2ly
Try `--no-discard-stderr' if option outputs to stderr
/home/hermann/lilypond-git/stepmake/stepmake/help2man-rules.make:26: recipe for target 'out/musicxml2ly.1' failed
make[1]: *** [out/musicxml2ly.1] Error 1
make[1]: Leaving directory '/home/hermann/lilypond-git/build/scripts'
/home/hermann/lilypond-git/stepmake/stepmake/generic-targets.make:6: recipe for target 'all' failed
make: *** [all] Error 2

John Gourlay - 2016-05-09

Thomas,

My local branch from which I created the repository branch builds, but I’ve been using make, not autogen. I’ll check out a clean copy of the branch and see how it works for me. In the meantime, you might try running musicxml2ly explicitly on any MusicXML file and see if you get a Python error that will shed some light on the matter. I’ve seen the same symptom that you’re seeing, and it has always meant that there is a Python error preventing musicxml2ly from running.

John

On May 8, 2016, at 5:16 AM, Thomas Morley thomas-morley@users.sf.net wrote:

John,
although your patch is beyond my depth I tried to do some checking. Thus I checked out your branch. But it failed make.
I then reran sh autogen.sh --noconfigure on it and ../configure in \build.
Though it still fails:

[...]
echo /home/hermann/lilypond-git/build/scripts/build/out/help2man
/home/hermann/lilypond-git/build/scripts/build/out/help2man
/home/hermann/lilypond-git/build/scripts/build/out/help2man out/musicxml2ly > out/musicxml2ly.1
help2man: can't get --help' info from out/musicxml2ly Try--no-discard-stderr' if option outputs to stderr
/home/hermann/lilypond-git/stepmake/stepmake/help2man-rules.make:26: recipe for target 'out/musicxml2ly.1' failed
make[1]: [out/musicxml2ly.1] Error 1
make[1]: Leaving directory '/home/hermann/lilypond-git/build/scripts'
/home/hermann/lilypond-git/stepmake/stepmake/generic-targets.make:6: recipe for target 'all' failed
make: [all] Error 2
[issues:#4751] Import Philomelos enhancements to musicxml2ly

Status: Started
Created: Sun Jan 24, 2016 02:27 AM UTC by John Gourlay
Last Updated: Sat May 07, 2016 09:22 PM UTC
Owner: John Gourlay

Import Philomelos enhancements to musicxml2ly from github.com/Philomelos/lilypond-musicxml2ly-dev. This includes the following:
- New command line option --transpose c TOPITCH.
- Added the sound tempo recognition for midi output.
- Added support for standalone sound elements.
- Added the --shift-meter option to make the music look faster/slower.
- Implemented recognition of stem values "up" and "down".
- Added support for ChordNames transposition.
- Added the command line options --tc / --tabclef [tab|moderntab]
to be able to switch between the two styles of the tab clef.
- Added transpose support for FretDiagrams.
- Allow only "moderntab" and "tab" for tab_clef value.
- Added the command line options --sn / --string-numbers [t|f] to activate
(default) | deactivate the string number stencil.
- Added conversion of elements to a separate FretBoards voice.
- No longer use the staff tuning option.
- Added --no-stem-direction option to ignore stem directions from MusicXML.
- Added <credit> elements recognition.
- Added page layout handling options.
- Fixed the issue with spaces/brackets in filename.
- Recognize the print-lyric attribute.
- Colored noteheads.
- Allow both <fermata>angled</fermata> and <fermata type="angled"></fermata>
and convert them correctly.
- Color-attribute ignored. Color of notehead and stem can be set with
hexadecimal strings. Color of stem is set even if
conversion_settings.convert_stem_direction is False.
- Include articulate.ly if options.midi == True.
- If a <instrument-sound>-tag is present, attempt to map its value to a
corresponding Lilypond midi-instrument and assign it to the correct staff.
- Automatically include the philomelos tagline in the header.
- Fix numerous bugs.</instrument-sound></credit>

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/testlilyissues/issues/4751/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John,
I'm used to build lilypond following
http://lilypond.org/doc/v2.19/Documentation/contributor/compiling-with-lilydev
If I nuke my build-directory and do all from scratch your branch successfully builds now.
Though this prevents keeping the test-baseline to run make check against, at least following the method in CG. Though my attempt to do regtest-comparision localy didn't work out.

Nevertheless, I took all files from /input/regression/musicxml and compiled them (most simple) with

for f in *.xml
do
    musicxml2ly "$f"
done

Two files returned problems:

musicxml2ly: Reading MusicXML from 32b-Articulations-Texts.xml ...
musicxml2ly: Converting to LilyPond expressions...
musicxml2ly: Output to `32b-Articulations-Texts.ly'
Traceback (most recent call last):
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 3269, in <module>
    main()
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 3263, in main
    voices = convert(filename, options)
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 3180, in convert
    print_voice_definitions(printer, part_list, voices)
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 2940, in print_voice_definitions
    voice.ly_voice.print_ly (printer)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 579, in print_ly
    MusicWrapper.print_ly (self, func)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 553, in print_ly
    self.element.print_ly (func)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 755, in print_ly
    e.print_ly (printer)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 546, in print_ly
    printer (self.ly_expression ())
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 131, in __call__
    self.dump (arg)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 139, in dump
    words = utilities.split_string_and_preserve_doublequoted_substrings(str)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/utilities.py", line 72, in split_string_and_preserve_doublequoted_substrings
    return list(lex)
  File "/usr/lib/python2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/usr/lib/python2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/usr/lib/python2.7/shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation

musicxml2ly: Reading MusicXML from 45f-Repeats-InvalidEndings.xml ...
musicxml2ly: Converting to LilyPond expressions...
musicxml2ly: Output to `45f-Repeats-InvalidEndings.ly'
Traceback (most recent call last):
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 3269, in <module>
    main()
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 3263, in main
    voices = convert(filename, options)
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 3180, in convert
    print_voice_definitions(printer, part_list, voices)
  File "/home/hermann/lilypond-git/build/out/bin/musicxml2ly", line 2940, in print_voice_definitions
    voice.ly_voice.print_ly (printer)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 579, in print_ly
    MusicWrapper.print_ly (self, func)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 553, in print_ly
    self.element.print_ly (func)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 755, in print_ly
    e.print_ly (printer)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 794, in print_ly
    self.music.print_ly (printer)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicexp.py", line 755, in print_ly
    e.print_ly (printer)
  File "/home/hermann/lilypond-git/build/out/lib/lilypond/current/python/musicxml2ly_conversion.py", line 41, in print_ly
    ly.warning(_("Encountered unprocessed marker %s\n") % self)
NameError: global name '_' is not defined

I hope I took the correct music2ly:

musicxml2ly --version
musicxml2ly (LilyPond) 2.19.41

Copyright (c) 2005--2016 by
    Han-Wen Nienhuys <hanwen@xs4all.nl>,
    Jan Nieuwenhuizen <janneke@gnu.org> and
    Reinhold Kainhofer <reinhold@kainhofer.com>
    Patrick L. Schmidt <pls@philomelos.net>

This program is free software.  It is covered by the GNU General Public
License and you are welcome to change it and/or distribute copies of it
under certain conditions.  Invoke as `lilypond --warranty' for more
information.

looks correct, though

John Gourlay - 2016-05-17

Thomas,

Thanks for continuing to look at this. Unfortunately, I discovered this morning that I can’t log into my lilydev virtual machine. Until I fix this problem I won’t be able to do any more work on the musicxml2ly issue.

I think you already noticed, but last week I pushed a commit to my development branch that fixed a fundamental flaw in my original patch and branch. The commit creates a file python/musicxml2ly_conversion.py, which was required by the existing file python/musicxml.py, and without which musicxml2ly would not run at all. (I overlooked this problem because I was doing my testing in a working directory that, without me noticing, contained an old copy of musicxml2ly_conversion.py.) The fact that musicxml2ly runs at all for you probably means that you’ve pulled this change, but maybe you should double check.

After I fixed that problem my next step was to verify that a clean build made from the branch works fully on all the regression tests. I wasn’t able to do that because “make doc” and “make test” would not work consistently for me. I saw messages about a similar problem from the person who does the weekly patch work, so I decided to wait for that problem to be solved before I continued my work. Now, I can’t even log into Debian. I’m not having fun.

John

On May 16, 2016, at 12:14 PM, Thomas Morley thomas-morley@users.sf.net wrote:

Two files returned problems:

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thomas Morley - 2016-05-19

John,
sorry to hear about your VM, hope you can fix it soon.
I've found musicxml2ly_conversion.py in my ~/lilypond-git/build/python/out, so this seems correct.
But as long as this issue is labeled Patch: needs_work you will not get much replies here.
With a working VM I'd first try to update your patch on Rietveld with the missing code.
For the regtest-comparision I'd ask on the devel-list. More likely to get a reply there.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

John Gourlay - 2016-05-26

Make additional changes to the new version of musicxml2ly imported
from the Philomelos project. The changes are also uploaded to the
branch dev/johngourlay/issue-4751. After these changes I can run the
following build commands without error:

make
make test-clean
make test
make check

The problems with the earlier patch were due to my misunderstanding of
the make test dependencies. I did not run make test-clean, and so I did
not see that some musicxml regression tests did not run.

http://codereview.appspot.com/297370043

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-06-01

Needs: -->

Patch: new --> review

Type: --> Enhancement
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-06-01

Passes make, make check and a full make doc

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-06-06

Patch: review --> countdown
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anonymous - 2016-06-06

Patch on countdown for June 9th.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Import Philomelos enhancements to musicxml2ly

Test issue tracker

Searches

Help

#4751 Import Philomelos enhancements to musicxml2ly

Discussion