#5721 lilypond-book: Rewrite processing of snippets

[Jonas Hahnfeld]

lilypond-book: Rewrite processing of snippets

The previous version relied on a stable hash() method, both for the
ordering in a set() and for the list's checksum. This probably worked
with Python 2 and up to Python 3.2, but later versions use a random
seed for invocations of hash(). This ensures different hash values for
subsequent invocations to make malicious attacks more difficult.

The new code uses hashlib.md5() and .hexdigest() instead which returns
a deterministic result across runs. It also sorts the snippets' names
which leads to more stable profiling results for 'make check'.
This change also tries to improve performance by writing snippets with
the same basename only once. Additionally it solves potential problems
if the build directory has the string '.ly' in its path.

http://codereview.appspot.com/555220043

[Anonymous]

Passes make, make check and a full make doc.

[Jonas Hahnfeld]

Address comments & revert temporary workaround in master

http://codereview.appspot.com/555220043

[Anonymous]

Passes make, make check and a full make doc.

[Jonas Hahnfeld]

snippet-names-*.ly -> .txt

http://codereview.appspot.com/555220043

[Jonas Hahnfeld]

Revert change to .txt

http://codereview.appspot.com/555220043

[Anonymous]

Passes make, make check and a full make doc.
(although I get all those cell counts back again, I assume this would be expected in this one case - ie. I am checking against a baseline that used .ly and now is using .txt)

[Jonas Hahnfeld]

Yes, profiling differences are expected for this change because the snippets are in a different order. However I had to revert the .txt change, it's back to .ly ;-)

[Anonymous]

Patch on countdown for Feb 6th.

[Anonymous]

Patch counted down - please push.

[Jonas Hahnfeld]

pushed to staging

commit a621f28387363d4c71b90e3a0d0686a5d2bedccf
Author:     Jonas Hahnfeld <hahnjo@hahnjo.de>
AuthorDate: Sun Feb 2 10:36:54 2020 +0100
Commit:     Jonas Hahnfeld <hahnjo@hahnjo.de>
CommitDate: Thu Feb 6 12:00:42 2020 +0100

    Issue 5721/2: Revert "Disable Python's hash randomization"

    Now that lilypond-book uses a real hash function, remove the workaround.
    This reverts commit 04a0dc897629d1606d12d54b5c0e15e9cd003652.

commit c2808fd480612a55ff8aec91480d9097ae05359b
Author:     Jonas Hahnfeld <hahnjo@hahnjo.de>
AuthorDate: Thu Jan 30 21:45:54 2020 +0100
Commit:     Jonas Hahnfeld <hahnjo@hahnjo.de>
CommitDate: Thu Feb 6 12:00:34 2020 +0100

    Issue 5721/1: lilypond-book: Rewrite processing of snippets

    The previous version relied on a stable hash() method, both for the
    ordering in a set() and for the list's checksum. This probably worked
    with Python 2 and up to Python 3.2, but later versions use a random
    seed for invocations of hash(). This ensures different hash values for
    subsequent invocations to make malicious attacks more difficult.

    The new code uses hashlib.md5() and .hexdigest() instead which returns
    a deterministic result across runs. It also sorts the snippets' names
    which leads to more stable profiling results for 'make check'.
    This change also tries to improve performance by writing snippets with
    the same basename only once. Additionally it solves potential problems
    if the build directory has the string '.ly' in its path.