openlilylib-user

[oll-user] 'snippets' Github interface idea

[Urs L. <ul...@op...>]

Hi folks,

I try for the first time to use the oll-user list for the purpose of 
discussion of the new snippets repository.

I suggest doing this in the future for two reasons:
- automatically include anybody who might be interested
   (including people one wouldn't have thought of in the first place)
- keeping the content available in the public mailing list archive

#########################

I had an idea to improve the web presentation of the snippets on Github:

Write a script that iterates over all snippets and generate a Markdown 
file to display the snippet in the Github interface.
I see two options:
- generate an accompanying .md file that can be clicked upon
- place all snippets in its dedicated subdirectory and create a 
README.md file

I would suggest the second solution although that will result in a large 
number of subdirectories.
This way as soon as one opens the directory the README is displayed 
automatically.

In the basic version the generated file would contain
- A heading and a general short introductory text
- A generated image (the example output of the compilable file)
   -> one more incentive to include good examples
- The complete code listing
   (unfortunately it isn't possible to provide code highlighting like in 
the blog posts)

In a later version one could (quite easily) extract meta information 
from the snippet and provide them in a formatted and structured display.

The script would be run locally every now and then by an admin.
Some issues to be considered:
- Try to determine snippets to be processed (i. e. only new or modified 
ones)
   recompiling snippets with differing LIlyPond versions would result in 
the images to be modified
   (i.e. detected as changes by Git)
- Determine appropriate LilyPond versions to compile the snippets

I would write such a script in Python.

What do you think?
Urs

Re: [oll-user] 'snippets' Github interface idea

[Janek W. <lem...@gm...>]

Hi all,

2013/9/10 Urs Liska <ul...@op...>

> Hi folks,
>
> I try for the first time to use the oll-user list for the purpose of
> discussion of the new snippets repository.
>
> I suggest doing this in the future for two reasons:
> - automatically include anybody who might be interested
>    (including people one wouldn't have thought of in the first place)
> - keeping the content available in the public mailing list archive
>

+1


> I had an idea to improve the web presentation of the snippets on Github:
>
> Write a script that iterates over all snippets and generate a Markdown
> file to display the snippet in the Github interface.
> I see two options:
> - generate an accompanying .md file that can be clicked upon
> - place all snippets in its dedicated subdirectory and create a
> README.md file
>
> I would suggest the second solution although that will result in a large
> number of subdirectories.
> This way as soon as one opens the directory the README is displayed
> automatically.
>

The more i think about this, the more i like it.  Having one additional
layer of directories seems to be a small price to pay for user-friendly
interface.  In fact, with such an interface our repository would be almost
as functional as LSR currently is.


> In the basic version the generated file would contain
> - A heading and a general short introductory text
> - A generated image (the example output of the compilable file)
>    -> one more incentive to include good examples
> - The complete code listing
>    (unfortunately it isn't possible to provide code highlighting like in
> the blog posts)
>

This could simply be a link to the snippet file, if we wanted to make
things simple.

Actually, we may be able to provide syntax highligting: as far as i know,
you can insert HTML into markdown, so we could use your html-export from
Frescobaldi to produce html code and paste it into the .md file!
...for some reason this doesn't work as expected.  I've added a markdown
file containing html code generated using your Frescobaldi export, and it
isn't coloredd.  I think that Github isn't respecting some aspect of html
formatting, maybe that's a bug.
See my attempt here:
https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md

In a later version one could (quite easily) extract meta information
> from the snippet and provide them in a formatted and structured display.
>

In fact, we might want to switch to markdown in snippet descriptions
because of this.

The script would be run locally every now and then by an admin.
>

Easy to do with a cron job.


> Some issues to be considered:
> - Try to determine snippets to be processed (i. e. only new or modified
> ones)
>    recompiling snippets with differing LIlyPond versions would result in
> the images to be modified
>    (i.e. detected as changes by Git)
>

This should be easy: just store the committish of the revision of the
snippet that was used for producing .md and .png files, and check if think
that having a .png file in the snippet directory.  If there were changes in
the snippet file since that commit, recompile.


> - Determine appropriate LilyPond versions to compile the snippets
>
> I would write such a script in Python.
>
> What do you think?
>

go for it!

best,
Janek
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Urs L. <ul...@op...>]

Am 13.09.2013 01:20, schrieb Janek Warchoł:
> Hi all,
>
> 2013/9/10 Urs Liska <ul...@op... <mailto:ul...@op...>>
>
>     I had an idea to improve the web presentation of the snippets on
>     Github:
>
>     Write a script that iterates over all snippets and generate a Markdown
>     file to display the snippet in the Github interface.
>     I see two options:
>     - generate an accompanying .md file that can be clicked upon
>     - place all snippets in its dedicated subdirectory and create a
>     README.md file
>
>     I would suggest the second solution although that will result in a
>     large
>     number of subdirectories.
>     This way as soon as one opens the directory the README is displayed
>     automatically.
>
>
> The more i think about this, the more i like it. Having one additional 
> layer of directories seems to be a small price to pay for 
> user-friendly interface.  In fact, with such an interface our 
> repository would be almost as functional as LSR currently is.
>
>     In the basic version the generated file would contain
>     - A heading and a general short introductory text
>     - A generated image (the example output of the compilable file)
>        -> one more incentive to include good examples
>     - The complete code listing
>        (unfortunately it isn't possible to provide code highlighting
>     like in
>     the blog posts)
>
>
> This could simply be a link to the snippet file, if we wanted to make 
> things simple.

I would prefer including the listing. This shouldn't be really 
complicated, and it would make the code readable in the context of the 
description. Opening .ly files in a browser usually isn't really 
exciting ...
>
> Actually, we may be able to provide syntax highligting: as far as i 
> know, you can insert HTML into markdown, so we could use your 
> html-export from Frescobaldi to produce html code and paste it into 
> the .md file!
> ...for some reason this doesn't work as expected.  I've added a 
> markdown file containing html code generated using your Frescobaldi 
> export, and it isn't coloredd.  I think that Github isn't respecting 
> some aspect of html formatting, maybe that's a bug.
> See my attempt here:
> https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md

As you noticed, I had tried the same before ;-)
Having written to Github is a good idea. Although I don't think it's a 
bug but rather an omission. Maybe even on purpose, e.g. for security 
reasons.

>
>     In a later version one could (quite easily) extract meta information
>     from the snippet and provide them in a formatted and structured
>     display.
>
>
> In fact, we might want to switch to markdown in snippet descriptions 
> because of this.
>
>     The script would be run locally every now and then by an admin.
>
>
> Easy to do with a cron job.

I would prefer doing it as a Git hook.
It should be possible to check which files have been affected by a 
commit and act upon this information.

>
>     Some issues to be considered:
>     - Try to determine snippets to be processed (i. e. only new or
>     modified
>     ones)
>        recompiling snippets with differing LIlyPond versions would
>     result in
>     the images to be modified
>        (i.e. detected as changes by Git)
>
>
> This should be easy: just store the committish of the revision of the 
> snippet that was used for producing .md and .png files, and check if 
> think that having a .png file in the snippet directory.  If there were 
> changes in the snippet file since that commit, recompile.

You're more experienced with Git than I am. But I think if we hook into 
the commit itself it would be easier and more reliable to retrieve 
information on the affected files themselves.

>
>     - Determine appropriate LilyPond versions to compile the snippets
>
>     I would write such a script in Python.
>
>     What do you think?
>
>
> go for it!

One day ...

Best
Urs
>
> best,
> Janek

-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Janek W. <lem...@gm...>]

2013/9/13 Urs Liska <ul...@op...>

>  Am 13.09.2013 01:20, schrieb Janek Warchoł:
>
 In the basic version the generated file would contain
>> - A heading and a general short introductory text
>> - A generated image (the example output of the compilable file)
>>    -> one more incentive to include good examples
>> - The complete code listing
>>    (unfortunately it isn't possible to provide code highlighting like in
>> the blog posts)
>>
>
>  This could simply be a link to the snippet file, if we wanted to make
> things simple.
>
> I would prefer including the listing. This shouldn't be really
> complicated, and it would make the code readable in the context of the
> description. Opening .ly files in a browser usually isn't really exciting
> ...
>

+1

>  Actually, we may be able to provide syntax highligting: as far as i
> know, you can insert HTML into markdown, so we could use your html-export
> from Frescobaldi to produce html code and paste it into the .md file!
> ...for some reason this doesn't work as expected.  I've added a markdown
> file containing html code generated using your Frescobaldi export, and it
> isn't coloredd.  I think that Github isn't respecting some aspect of html
> formatting, maybe that's a bug.
>  See my attempt here:
>
> https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md
>
>
> As you noticed, I had tried the same before ;-)
> Having written to Github is a good idea. Although I don't think it's a bug
> but rather an omission. Maybe even on purpose, e.g. for security reasons.
>

As for now i was told to report this to pygments:
http://bitbucket.org/birkenfeld/pygments-main/issues

>   In a later version one could (quite easily) extract meta information
>> from the snippet and provide them in a formatted and structured display.
>>
>
>  In fact, we might want to switch to markdown in snippet descriptions
> because of this.
>
>  The script would be run locally every now and then by an admin.
>>
>
>  Easy to do with a cron job.
>
>  I would prefer doing it as a Git hook.
> It should be possible to check which files have been affected by a commit
> and act upon this information.
>

I suppose that It would have to be done as a post-fetch hook, and we would
probably need cron jobs anyway to ensure that some of the admins fetch
regularly.

>   Some issues to be considered:
>> - Try to determine snippets to be processed (i. e. only new or modified
>> ones)
>>    recompiling snippets with differing LIlyPond versions would result in
>> the images to be modified
>>    (i.e. detected as changes by Git)
>>
>
>  This should be easy: just store the committish of the revision of the
> snippet that was used for producing .md and .png files, and check if think
> that having a .png file in the snippet directory.  If there were changes in
> the snippet file since that commit, recompile.
>
>
> You're more experienced with Git than I am. But I think if we hook into
> the commit itself it would be easier and more reliable to retrieve
> information on the affected files themselves.
>

I haven't used git hooks yet, but as far as i understand it to use a
post-commit hook everyone needs to configure this hook in their
repositories, and probably it wouldn't work with github's web interface.
So, instead of using the post-commit hook we would run the script as
post-fetch hook.

best,
Janek
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Janek W. <lem...@gm...>]

2013/9/13 Janek Warchoł <lem...@gm...>

> 2013/9/13 Urs Liska <ul...@op...>
>
>>  Am 13.09.2013 01:20, schrieb Janek Warchoł:
>>
>   Actually, we may be able to provide syntax highligting: as far as i
>> know, you can insert HTML into markdown, so we could use your html-export
>> from Frescobaldi to produce html code and paste it into the .md file!
>> ...for some reason this doesn't work as expected.  I've added a markdown
>> file containing html code generated using your Frescobaldi export, and it
>> isn't coloredd.  I think that Github isn't respecting some aspect of html
>> formatting, maybe that's a bug.
>>  See my attempt here:
>>
>> https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md
>>
>>
>> As you noticed, I had tried the same before ;-)
>> Having written to Github is a good idea. Although I don't think it's a
>> bug but rather an omission. Maybe even on purpose, e.g. for security
>> reasons.
>>
>
> As for now i was told to report this to pygments:
> http://bitbucket.org/birkenfeld/pygments-main/issues
>

https://bitbucket.org/birkenfeld/pygments-main/issue/904/lilypond-support
I've also asked Github about broken HTML colors in markdown display.

best,
Janek
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Urs L. <ul...@op...>]

Am 13.09.2013 17:12, schrieb Janek Warchoł:
>
>
>
> 2013/9/13 Janek Warchoł <lem...@gm... 
> <mailto:lem...@gm...>>
>
>     2013/9/13 Urs Liska <ul...@op... <mailto:ul...@op...>>
>
>         Am 13.09.2013 01:20, schrieb Janek Warchoł:
>
>>         Actually, we may be able to provide syntax highligting: as
>>         far as i know, you can insert HTML into markdown, so we could
>>         use your html-export from Frescobaldi to produce html code
>>         and paste it into the .md file!
>>         ...for some reason this doesn't work as expected.  I've added
>>         a markdown file containing html code generated using your
>>         Frescobaldi export, and it isn't coloredd.  I think that
>>         Github isn't respecting some aspect of html formatting, maybe
>>         that's a bug.
>>         See my attempt here:
>>         https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md
>
>         As you noticed, I had tried the same before ;-)
>         Having written to Github is a good idea. Although I don't
>         think it's a bug but rather an omission. Maybe even on
>         purpose, e.g. for security reasons.
>
>
>     As for now i was told to report this to pygments:
>     http://bitbucket.org/birkenfeld/pygments-main/issues
>
>
> https://bitbucket.org/birkenfeld/pygments-main/issue/904/lilypond-support

Does that mean GitHub doesn't support formatted HTML code inside markdown?
Because what pygments does is something completely different, isn't it 
(although some generic highlighting using a widely used library like 
pygments would be a good thing in its own right).

> I've also asked Github about broken HTML colors in markdown display.

What are you refering to with that?

Urs
>
> best,
> Janek

-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Janek W. <lem...@gm...>]

2013/9/13 Urs Liska <ul...@op...>

>  Am 13.09.2013 17:12, schrieb Janek Warchoł:
>
>
> 2013/9/13 Janek Warchoł <lem...@gm...>
>
>>   2013/9/13 Urs Liska <ul...@op...>
>>
>>>  Am 13.09.2013 01:20, schrieb Janek Warchoł:
>>>
>>           Actually, we may be able to provide syntax highligting: as far
>>> as i know, you can insert HTML into markdown, so we could use your
>>> html-export from Frescobaldi to produce html code and paste it into the .md
>>> file!
>>> ...for some reason this doesn't work as expected.  I've added a markdown
>>> file containing html code generated using your Frescobaldi export, and it
>>> isn't coloredd.  I think that Github isn't respecting some aspect of html
>>> formatting, maybe that's a bug.
>>>  See my attempt here:
>>>
>>> https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md
>>>
>>>
>>>  As you noticed, I had tried the same before ;-)
>>> Having written to Github is a good idea. Although I don't think it's a
>>> bug but rather an omission. Maybe even on purpose, e.g. for security
>>> reasons.
>>>
>>
>>  As for now i was told to report this to pygments:
>> http://bitbucket.org/birkenfeld/pygments-main/issues
>>
>
> https://bitbucket.org/birkenfeld/pygments-main/issue/904/lilypond-support
>
>
> Does that mean GitHub doesn't support formatted HTML code inside markdown?
> Because what pygments does is something completely different, isn't it
> (although some generic highlighting using a widely used library like
> pygments would be a good thing in its own right).
>

Sorry, i've indeed mixed two things.
When i talked with GitHub support about syntax highlighting in general, and
the possibility of GitHub highlighting Lily sources automatically (i.e.
when *.ly files are viewed on github website), i was advised to add an
issue to Pygments.  And so i did.

>    I've also asked Github about broken HTML colors in markdown display.
>
>  What are you refering to with that?
>

To your previous paragraph :) I.e. apart from asking Pygments to support
LilyPond, i've asked Github what's up with Github's support for formatted
HTML code inside markdown.

best,
Janek
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Urs L. <ul...@op...>]

Am 13.09.2013 18:03, schrieb Janek Warchoł:
> 2013/9/13 Urs Liska <ul...@op... <mailto:ul...@op...>>
>
>     Am 13.09.2013 17:12, schrieb Janek Warchoł:
>>
>>     2013/9/13 Janek Warchoł <lem...@gm...
>>     <mailto:lem...@gm...>>
>>
>>         2013/9/13 Urs Liska <ul...@op...
>>         <mailto:ul...@op...>>
>>
>>             Am 13.09.2013 01:20, schrieb Janek Warchoł:
>>
>>>             Actually, we may be able to provide syntax highligting:
>>>             as far as i know, you can insert HTML into markdown, so
>>>             we could use your html-export from Frescobaldi to
>>>             produce html code and paste it into the .md file!
>>>             ...for some reason this doesn't work as expected.  I've
>>>             added a markdown file containing html code generated
>>>             using your Frescobaldi export, and it isn't coloredd. I
>>>             think that Github isn't respecting some aspect of html
>>>             formatting, maybe that's a bug.
>>>             See my attempt here:
>>>             https://github.com/openlilylib/snippets/blob/markdown-test/notation-snippets/scaling-stencils.md
>>
>>             As you noticed, I had tried the same before ;-)
>>             Having written to Github is a good idea. Although I don't
>>             think it's a bug but rather an omission. Maybe even on
>>             purpose, e.g. for security reasons.
>>
>>
>>         As for now i was told to report this to pygments:
>>         http://bitbucket.org/birkenfeld/pygments-main/issues
>>
>>
>>     https://bitbucket.org/birkenfeld/pygments-main/issue/904/lilypond-support
>
>     Does that mean GitHub doesn't support formatted HTML code inside
>     markdown?
>     Because what pygments does is something completely different,
>     isn't it (although some generic highlighting using a widely used
>     library like pygments would be a good thing in its own right).
>
>
> Sorry, i've indeed mixed two things.
> When i talked with GitHub support about syntax highlighting in 
> general, and the possibility of GitHub highlighting Lily sources 
> automatically (i.e. when *.ly files are viewed on github website), i 
> was advised to add an issue to Pygments.  And so i did.
>
>>     I've also asked Github about broken HTML colors in markdown display.
>     What are you refering to with that?
>
>
> To your previous paragraph :) I.e. apart from asking Pygments to 
> support LilyPond, i've asked Github what's up with Github's support 
> for formatted HTML code inside markdown.

So do I see correctly?

a)
You asked for displaying .ly files and were forwarded to pygments.
This should be doable, but it looks quite crowded there, 51 open pull 
requests alone ...

b)
You asked for being able to used styled HTML inside Markdown.
And this is still open?
Or did they point you to pygments too with that question?

Urs
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Janek W. <lem...@gm...>]

2013/9/13 Urs Liska <ul...@op...>

> So do I see correctly?
>
> a)
> You asked for displaying .ly files and were forwarded to pygments.
> This should be doable, but it looks quite crowded there, 51 open pull
> requests alone ...
>

yes


> b)
> You asked for being able to used styled HTML inside Markdown.
> And this is still open?
>

yes.
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] 'snippets' Github interface idea

[Janek W. <lem...@gm...>]

2013/9/13 Janek Warchoł <lem...@gm...>

>
> 2013/9/13 Urs Liska <ul...@op...>
>
>> b)
>> You asked for being able to used styled HTML inside Markdown.
>> And this is still open?
>>
>
> yes.
>

Reply from GitHub:

2013/9/13 Petros Amiridis <su...@gi...>:
> Hi Janek,
>
>> What about broken(?) display of HTML formatting in Markdown files?
>
> Fully featured HTML code can't survive our sanitizing filters. You can
have some HTML in a markdown file, but some tags will not work. For example
your <span> tags will all get stripped, but a <strong> tag can survive.

So Pygment seems to be our only hope.

Janek
-------------- next part --------------
An HTML attachment was scrubbed...