openlilylib-user

[oll-user] Repository organization

[Urs L. <ul...@op...>]

Hi,

this is the first email separating issues from a recent thread
(http://sourceforge.net/mailarchive/forum.php?thread_name=5147033D.9030301%40openlilylib.org&forum_name=openlilylib-user)
into separate topics.
I'd be glad if we could keep them separate as much as possible ;-)

Joe is highly in favor of separating our stuff into separate Git submodules.
The most important statements are possibly the following (you can find 
more in the thread):

> but I don't see the point in having
> one-repo-to-rule-them-all_unless_  done via submodules -- the separation into 3
> different projects is quite correct IMO, as each part is clearly independently
> useful (or at least, such dependencies as exist are one-way only).

> I think you need to think of it like this: users who only want to_use_  one or
> more of the parts may operate off archived releases, but what about users who
> only want to_work_  on one of the parts?  It's really not clear to me why
> someone who wants to work solely on lilyglyphs (say) should have to pull in
> OLLib or the manuals.
Basically I fully agree with that POV, as the sub-projects really are quite independent conceptually, and I agree that the interdependence of the current design is based on my perception of the 'whole' which is far from being carved in stone.

However, I'm still somewhat insecure about how the submodule approach is actually manageable.
 From trying to understand the docs and reading blog posts I have two questions:

1)
A submodule isn't just a Git repository living in a subdirectory. Instead, a submodule points to another remote repository, so this seems to mean that we have (e.g.) lilyglyphs as a subdirectory/submodule within openlilylib _and_ as  standalone repository.
I understand this set-up for the usual examples of a third-party library, but I don't really see how this is natural for our case.

2)
I have read that it is comparably 'simple' to get the repo/submodule constellation out of sync and even corrupted (if one part points to a commit that the other part hasn't pushed etc.).
How complicated does a working policy have to prevent this? And how complicated is it to recover from such situations?
Instead of an explanation I would also accept statements like "I know this works smoothly, and you'll see it when it is running".

If that's actually manageable, I'd happily accept to split the main repository in as many submodules as seem appropriate. But I'd be glad if someone could guide me through the process of doing so without me having to learn everything from scratch.

Best
Urs

Re: [oll-user] Repository organization

[Joseph R. W. <jos...@we...>]

On 04/05/2013 05:01 PM, Urs Liska wrote:
> Basically I fully agree with that POV, as the sub-projects really are quite independent conceptually, and I agree that the interdependence of the current design is based on my perception of the 'whole' which is far from being carved in stone.

The other key reason to separate is about maintenance and commit right
bottlenecks.  By having everything in one big repo, you wind up either having to
give lots of people commit rights (not very safe), or you have to channel all
contributions through a few people.

By splitting things into separately-versioned projects, you can have different
people maintaining different parts, so a small number of committers on a
per-project basis but a large number of people involved overall.

> However, I'm still somewhat insecure about how the submodule approach is actually manageable.
>  From trying to understand the docs and reading blog posts I have two questions:
> 
> 1)
> A submodule isn't just a Git repository living in a subdirectory. Instead, a submodule points to another remote repository, so this seems to mean that we have (e.g.) lilyglyphs as a subdirectory/submodule within openlilylib _and_ as  standalone repository.
> I understand this set-up for the usual examples of a third-party library, but I don't really see how this is natural for our case.
> 
> 2)
> I have read that it is comparably 'simple' to get the repo/submodule constellation out of sync and even corrupted (if one part points to a commit that the other part hasn't pushed etc.).
> How complicated does a working policy have to prevent this? And how complicated is it to recover from such situations?
> Instead of an explanation I would also accept statements like "I know this works smoothly, and you'll see it when it is running".

Can't speak to (2) as I have never encountered it.  But if you are concerned,
the simplest way is just to give up the idea of one-repo-to-rule-them-all and
have a few different repositories to cover different parts of the project.

It's zero added complexity for users of the project, as one can always bundle
everything together in one big .tar.gz file for user download; it adds some
complexity for developers/contributors but also reduces complexity in other ways.

So, I would start by modularizing the project appropriately into different repos
without worrying about submodules, and think about introducing them later if
that seems to be necessary.

Re: [oll-user] Repository organization

[Janek W. <lem...@gm...>]

Time to catch up with some things..

2013/4/5 Joseph Rushton Wakeling <jos...@we...>

> But if you are concerned,
> the simplest way is just to give up the idea of one-repo-to-rule-them-all
> and
> have a few different repositories to cover different parts of the project.
>
> It's zero added complexity for users of the project, as one can always
> bundle
> everything together in one big .tar.gz file for user download; it adds some
> complexity for developers/contributors but also reduces complexity in
> other ways.
>
> So, I would start by modularizing the project appropriately into different
> repos
> without worrying about submodules, and think about introducing them later
> if
> that seems to be necessary.
>

This sounds reasonable.  As long as we don't have more than a few repos in
a flat directory structure, that shouldn't be very complicated to execute.
If we encounter problems, we can fix them in the future.
Janek
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] Repository organization

[Urs L. <ul...@op...>]

Am 12.04.2013 14:13, schrieb Janek Warchoł:
> Time to catch up with some things..
>
> 2013/4/5 Joseph Rushton Wakeling <jos...@we...>
>
>> But if you are concerned,
>> the simplest way is just to give up the idea of one-repo-to-rule-them-all
>> and
>> have a few different repositories to cover different parts of the project.
>>
>> It's zero added complexity for users of the project, as one can always
>> bundle
>> everything together in one big .tar.gz file for user download; it adds some
>> complexity for developers/contributors but also reduces complexity in
>> other ways.
>>
>> So, I would start by modularizing the project appropriately into different
>> repos
>> without worrying about submodules, and think about introducing them later
>> if
>> that seems to be necessary.
>>
> This sounds reasonable.  As long as we don't have more than a few repos in
> a flat directory structure, that shouldn't be very complicated to execute.
> If we encounter problems, we can fix them in the future.
> Janek
I also thought this is reasonable. We should probably have separate 
repos for
- the LilyPond library
- common LaTeX stuff (that is necessary to compile documents)
- tutorials
- web site
- lilyglyphs
- musicexamples
- (lilydoc [if it should come into existence ...])

But it seems not so easy to separate the repositories regarding their 
history.

I tried to rip off a musicexamples repository but failed so far.
First I made a copy of the existing repo and restored the old history 
that had been stuffed away in the archive tag. This worked quite well, 
and I'm very happy I did that.

Then I tried to extract the history for the musicexamples directory with 
git filter-branch --directory-filter, and this didn't work:
- filter-branch --directory-filter works only for files that are in that 
dir _currently_
commits with files that have been moved into at some time are not included
-> the oldest transfered commit was consequently titled 'move to 
musicexamples dir' ,-)
- reason for that is:
- filter-branch pulls the directory to the root level
- if a file had been moved from a directory _above_ the target of 
--directory-filter
it would end up being considered as outside the git repository

My research regarding the other filters didn't turn up a solution, at 
least none that I'd realized/understood.

I could log the history of the files with --follow which would include 
the history across renames. But this works only for singe files, not a 
directory.

An approach I could imagine is to recursive loop over all files in the 
directory and store all commits for each one, then merge these lists to 
one to remove duplicates, finally apply them all, maybe with cherry-pick 
on a new branch.
Of course this would be rather simple for example with Python, but I 
have to admit I'm not quite ready for such a thing.
And I don't quite see the implications of different branches for this 
approach yet.

So it remains open:
Any ideas how to deal with the situation?

The simplest solution would be:
- Just remove all files except the ones needed for the new repo with a 
regular commit
- Move all remaining files to the desired new location in the directory 
structure, also with a regular commit.
Absolutely painless process, but of course each repo would carry the 
whole history.

Second simplest solution:
- Add only the desired files to the new repos
- Keep the existing repo as the only instance of the history.

I don't like both.

Urs
> -------------- next part --------------
> An HTML attachment was scrubbed...
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> openlilylib-user mailing list
> ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openlilylib-user

Re: [oll-user] Repository organization

[Janek W. <lem...@gm...>]

2013/4/12 Urs Liska <ul...@op...>

> So it remains open:
> Any ideas how to deal with the situation?
>
> The simplest solution would be:
> - Just remove all files except the ones needed for the new repo with a
> regular commit
> - Move all remaining files to the desired new location in the directory
> structure, also with a regular commit.
> Absolutely painless process, but of course each repo would carry the
> whole history.
>
> Second simplest solution:
> - Add only the desired files to the new repos
> - Keep the existing repo as the only instance of the history.
>
> I don't like both.
>

What about: use --filter-branch to harvest as much history as possible, and
stop at that (keeping the existing repo as an archive of the history)?
It's not a very good solution, but i'm not sure if doing more is worth the
trouble.

I think only you can judge this, (as i've obviously had fallen behind with
the project): is the history important enough?  Were there many situations
where having current history significantly helped you?  One thing comes to
my mind: since the early history is quite tangled, it may not be very
helpful anyway in a few weeks/months from now.

best,
Janek
-------------- next part --------------
An HTML attachment was scrubbed...

Re: [oll-user] Repository organization

[Urs L. <ul...@op...>]

Am Freitag, den 12.04.2013, 19:18 +0200 schrieb Janek Warchoł:
> 
> 2013/4/12 Urs Liska <ul...@op...>
>         So it remains open:
>         Any ideas how to deal with the situation?
>         
>         The simplest solution would be:
>         - Just remove all files except the ones needed for the new
>         repo with a
>         regular commit
>         - Move all remaining files to the desired new location in the
>         directory
>         structure, also with a regular commit.
>         Absolutely painless process, but of course each repo would
>         carry the
>         whole history.
>         
>         Second simplest solution:
>         - Add only the desired files to the new repos
>         - Keep the existing repo as the only instance of the history.
>         
>         I don't like both.
> 
> 
> What about: use --filter-branch to harvest as much history as
> possible, and stop at that (keeping the existing repo as an archive of
> the history)?
> 
> It's not a very good solution, but i'm not sure if doing more is worth
> the trouble.
> 
> I think only you can judge this, (as i've obviously had fallen behind
> with the project): is the history important enough?  Were there many
> situations where having current history significantly helped you?  One
> thing comes to my mind: since the early history is quite tangled, it
> may not be very helpful anyway in a few weeks/months from now.

Probably you're right. Maybe I just lack the freedom of mind to let that
history go ...
(as one experienced musicologist told me: be careful with your critical
remarks. Don't write too many of them, even as the first sketch: Once
you have something written down you don't want to drop that anymore ;-)

Maybe that's a good compromise. I'll wait a few days (to see if somebody
comes up with another idea), and give it a try.

Urs
> 
> 
> best,
> 
> Janek
> 

Re: [oll-user] Repository organization

[Urs L. <ul...@op...>]

Am Freitag, den 12.04.2013, 20:54 +0200 schrieb Urs Liska:
> Am Freitag, den 12.04.2013, 19:18 +0200 schrieb Janek Warchoł:
> > 
> > 2013/4/12 Urs Liska <ul...@op...>
> >         So it remains open:
> >         Any ideas how to deal with the situation?
> >         
> > 
> > What about: use --filter-branch to harvest as much history as
> > possible, and stop at that (keeping the existing repo as an archive of
> > the history)?
> > 

> Maybe that's a good compromise. I'll wait a few days (to see if somebody
> comes up with another idea), and give it a try.
> 
Seems I'm the one myself ;-)
I realized that at least for the lilyglyph and musicexamples
repositories I haven't committed anything after merging the repos. So I
can just restore them :-)
- OLLib doesn't have any real history so far
- tutorials don't need a real history so far
- web doesn't either
- LaTeX stuff could be done the way you suggested.

Urs

> Urs
> > 
> > 
> > best,
> > 
> > Janek
> > 
> 
> 
> 
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> openlilylib-user mailing list
> ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openlilylib-user

Re: [oll-user] Repository organization

[Janek W. <lem...@gm...>]

2013/4/14 Urs Liska <ul...@op...>

> Am Freitag, den 12.04.2013, 20:54 +0200 schrieb Urs Liska:
> > Maybe that's a good compromise. I'll wait a few days (to see if somebody
> > comes up with another idea), and give it a try.
>
> Seems I'm the one myself ;-)
>

;)


> I realized that at least for the lilyglyph and musicexamples
> repositories I haven't committed anything after merging the repos. So I
> can just restore them :-)
> - OLLib doesn't have any real history so far
> - tutorials don't need a real history so far
> - web doesn't either
> - LaTeX stuff could be done the way you suggested.
>

ok, good luck!
Janek
-------------- next part --------------
An HTML attachment was scrubbed...