Pootle, Virtaal & The Translate Toolkit

On Do, 2007-06-14 at 05:55 +0200, lars wrote:
> Hi,
> 
> I submitted a patch proposal quite a long time ago


And several in the last day :-)

Thank you for the contributions!  I won't be able to look at them right
now, as I'm preparing to move house, but I hope to give them some
attention still this week.

Just to clarify, please say if I understand all of this correctly: 
 * SVN is tested, but not yet CVS and darcs (and yes, the _darcs is
strange to me too... I guess there is just one _darcs directory per
checkout)
 * all baseurl changes are tested, except for pagelayout.py lines 175
and 185 
 * The path handling is tested (platforms?)

I don't mind testing it all, it will just help me to understand where to
focus my attention :-)

The version control changes are perhaps big for a bugfix release, and
I'm also wondering if we shouldn't move it to the toolkit for the
benefit of future Pootling versions. But I'll still give it some
thought.

Thanks again!
Friedel

Hi,

> Just to clarify, please say if I understand all of this correctly:=20
>  * SVN is tested, but not yet CVS and darcs
yes

>  * all baseurl changes are tested, except for pagelayout.py lines 175
> and 185
yes - all baseurl changes are tested
as I did not know how to trigger the appropriate conditions for line 175 and
185, I did not change these lines
=20
>  * The path handling is tested (platforms?)
it is superficially tested on linux
These changes are simple and small - the only problem could be, that I
misinterpreted a URL (which is platform independent) for a filename. It wou=
ld be
good to check this again. Otherwise it could lead to problems on non-unix
platforms.=20
=46rom my point of understanding, I changed only filenames, so there should b=
e no
problem.

> The version control changes are perhaps big for a bugfix release,
yes, I think so, too

> and I'm also wondering if we shouldn't move it to the toolkit for the
> benefit of future Pootling versions. But I'll still give it some
> thought.
oh - I never thought about that. Sounds interesting ...

regards,
Lars

On Do, 2007-06-14 at 05:55 +0200, lars wrote:
> Hi,
> 
> I submitted a patch proposal quite a long time ago
> (12th of January 02007 -
> http://sourceforge.net/mailarchive/forum.php?thread_name=20070112043558.080c53b3%40erker.wg&forum_name=translate-pootle)
> regarding the version control support of pootle.
> At that time the patch caused problems as it required another python module
> (pysvn).
> Finally (sorry for the delay), I took the time to remove this dependency.
> The only intermediate change of trunk's versioncontrol.py was the implementation
> of darcs support. I integrated this, too.
> 
> The attached patch does the following:
> 1) allow to _symlink_ version controlled files (thus it is possible for
> projects to use a completely arbitrary directory layout without loosing pootle's
> version control feature) - read the previous thread for more details
> 2) use objects instead of plain functions for the handling of different version
> control systems - this should ease the work of adding more version control
> support (and it simplifies the code)
> 
> The patch successfully works regarding updates/commits with subversion. More
> tests for CVS and darcs should be necessary.
> 
> Do you think, that the patch goes into the right direction?
> 
> There is still some stuff missing, like:
> - avoid to use the shell for executing commands (by using arrays instead of
> strings for popen2)
> - replace shell commands with python functions (e.g. "mv")
> - replace unix-only pieces (e.g.: replace "/" with os.path.sep)
> - cleaning up the version control interface
> These things should get fixed afterwards.
> 
> regards,
> Lars

I just started looking at the patch now. I think I will probably commit
it quite soon, but here are just a few comments:
 * There are no users of the functions hascvs, hassvn, and hasdarcs -
can we drop them?
 * After moving to the toolkit, would it be meaningful to split this
into separate files? Perhaps even a dedicated directory under
storage/versioncontrol?
 * I will make some cosmetic changes, but I see things like the error
messages were not consistent before the change, so nothing against you,
Lars :-)

F

Hi,

>  * There are no users of the functions hascvs, hassvn, and hasdarcs -
> can we drop them?
yes - that sounds good

>  * After moving to the toolkit, would it be meaningful to split this
> into separate files? Perhaps even a dedicated directory under
> storage/versioncontrol?
yes, this would improve the structure.
But I am not really sure, if version control in general should be integrated in
this file base way, as it is now. I think, that a item based approach would be
more reasonable and should better fit to the (maybe) future database storage
backend (http://translate.sourceforge.net/wiki/pootle/metadata). As long as the
current code is only part of pootle, it is quite easy to change its interface.
As a part of the translate toolkit, it would be much harder to change it, I
guess.
But maybe the 'metadata' document is outdated anyway?
In short: yes - it should be splitted.

>  * I will make some cosmetic changes, but I see things like the error
> messages were not consistent before the change
yes - I wanted to keep the changes small and wait for a first "yes" or "no" ...

regards,
Lars

On Do, 2007-06-21 at 21:04 +0200, lars wrote:
> Hi,
> 
> >  * There are no users of the functions hascvs, hassvn, and hasdarcs -
> > can we drop them?
> yes - that sounds good
> 
> >  * After moving to the toolkit, would it be meaningful to split this
> > into separate files? Perhaps even a dedicated directory under
> > storage/versioncontrol?
> yes, this would improve the structure.
> But I am not really sure, if version control in general should be integrated in
> this file base way, as it is now. I think, that a item based approach would be
> more reasonable and should better fit to the (maybe) future database storage
> backend (http://translate.sourceforge.net/wiki/pootle/metadata). As long as the
> current code is only part of pootle, it is quite easy to change its interface.
> As a part of the translate toolkit, it would be much harder to change it, I
> guess.
> But maybe the 'metadata' document is outdated anyway?
> In short: yes - it should be splitted.

That metatdata document is quite old, yes.  I don't think I understand
what you mean with an item based approach. Can you expand a bit on that,
please? The only major shortcoming I currently see in the version
control functionality of Pootle is the ability to commit / update entire
directories/projects at a time. I don't know to what extent we would
have to extend the versioncontrol.py for that. 

> 
> >  * I will make some cosmetic changes, but I see things like the error
> > messages were not consistent before the change
> yes - I wanted to keep the changes small and wait for a first "yes" or "no" ...
> 
> regards,
> Lars

Obviously it is a yes :-)  We'll just short out the details now. Please
share your ideas of how to meaningfully move this to the toolkit and
what the cleanest organisation would be.

F

Na Tue, Jun 19, 2007 at 10:10:24AM +0200, F Wolff <friedel@...=
> pisal(a):
> I don't know if you are subscribed to the Pootle mailing list, so I'm
> forwarding this mail to you.

thanks, i was not subscribed

> Lars redid the current implementation of
> the version control to improve several things, but did not yet test it
> with CVS or Darcs.  Will you be able to test this for us and comment on
> it?

sure, i'll test it soon. originally i wanted to write you when i did the
test, but i don't want you to think that i'm dead :)

> +  ## TODO: could any user of darcs explain this path?
> +  MARKER_DIR =3D "../../../_darcs"

yes, though this variable name is quite misleading. when you record to
darcs, it records (=3D=3D commits) the difference between the pristine tree
(under /_darcs/pristine - or _darcs/current for older versions) with the
working directory (everything outside _darcs)

so we need to reach the _darcs dir from po/project/language. that will
result in ../../../_darcs

VMiklos

--=20
developer of Frugalware Linux - http://frugalware.org

Hi,

> > >  * After moving to the toolkit, would it be meaningful to split this
> > > into separate files? Perhaps even a dedicated directory under
> > > storage/versioncontrol?
> > yes, this would improve the structure.
> > But I am not really sure, if version control in general should be
> > integrated in this file base way, as it is now. I think, that a item based
> > approach would be more reasonable and should better fit to the (maybe)
> > future database storage backend
> > (http://translate.sourceforge.net/wiki/pootle/metadata). As long as the
> > current code is only part of pootle, it is quite easy to change its
> > interface. As a part of the translate toolkit, it would be much harder to
> > change it, I guess. But maybe the 'metadata' document is outdated anyway?
> > In short: yes - it should be splitted.
> 
> That metatdata document is quite old, yes.  I don't think I understand
> what you mean with an item based approach. Can you expand a bit on that,
> please?

For me, an "item" would be a single translated string (and possible meta data).
Right now the revision control interface does not care for indivual items, but
can only handle complete files. Thus any changes made through the web interface
are thrown away during an update from the version control repository. The same
goes for the other direction: intermediate changes in the version control
repository are (theoretically) ignored and overwritten by the changes
made through the web interface when committing.

In the end, this boils down to resolving conflicts, I guess.

I could imagine, that the web interface should offer the following options for
any conflicting item during revision control synchronization (update or commit):
A) keep RCS (revision control system) translation; turn the local translation
into a suggestion
B) keep RCS translation; throw away the local translation
C) keep the local translation; turn the RCS translation into a suggestion
D) keep the local translation; throw away the RCS translation

I would think, that similar conflicts do/will also arise in other situations
as well:
- two translators submitting a translation via the web interface within a short
timeframe - for now the one who submits later is the "winner" and will never
know about the translation, that was done before
- not yet existing: synchronization with other translation sources (offline
translation editors, other translation servers (like launchpad),
translations-by-mail, ...)

I do not have a real plan yet, how this should be implemented. But I would be
happy to work on it after some discussions (just in case, this is perceived as
a to-be-solved problem by the developers).

Anyway, I will move versioncontrol.py to a seperate directory and split it into
different files for different revision control systems. Then it should be easy
to move it to the toolkit, if necessary.

> The only major shortcoming I currently see in the version
> control functionality of Pootle is the ability to commit / update entire
> directories/projects at a time. I don't know to what extent we would
> have to extend the versioncontrol.py for that. 
the current code interface should be sufficient for that: I will do this in the
first week of July. But I am not sure where to integrate this into the web
interface.

Lars

On Sa, 2007-06-23 at 04:53 +0200, lars wrote:
> Hi,
> 
> > > >  * After moving to the toolkit, would it be meaningful to split this
> > > > into separate files? Perhaps even a dedicated directory under
> > > > storage/versioncontrol?
> > > yes, this would improve the structure.
> > > But I am not really sure, if version control in general should be
> > > integrated in this file base way, as it is now. I think, that a item based
> > > approach would be more reasonable and should better fit to the (maybe)
> > > future database storage backend
> > > (http://translate.sourceforge.net/wiki/pootle/metadata). As long as the
> > > current code is only part of pootle, it is quite easy to change its
> > > interface. As a part of the translate toolkit, it would be much harder to
> > > change it, I guess. But maybe the 'metadata' document is outdated anyway?
> > > In short: yes - it should be splitted.
> > 
> > That metatdata document is quite old, yes.  I don't think I understand
> > what you mean with an item based approach. Can you expand a bit on that,
> > please?
> 
> For me, an "item" would be a single translated string (and possible meta data).
> Right now the revision control interface does not care for indivual items, but
> can only handle complete files. Thus any changes made through the web interface
> are thrown away during an update from the version control repository. The same
> goes for the other direction: intermediate changes in the version control
> repository are (theoretically) ignored and overwritten by the changes
> made through the web interface when committing.

There is already code to handle these situations. A version control
conflict could cause an unparsable file, so we have to handle it for the
users.

> 
> In the end, this boils down to resolving conflicts, I guess.
> 
> I could imagine, that the web interface should offer the following options for
> any conflicting item during revision control synchronization (update or commit):
> A) keep RCS (revision control system) translation; turn the local translation
> into a suggestion

This is the current behaviour. We always consider version control to be
the authority.

Some notes are here:
http://translate.sourceforge.net/wiki/pootle/version_control

We might want to look at the detail to ensure that it is exactly what we
want, but I know people are already using this feature. The darcs
support comes from the people at Frugalware and their progress can be
viewed here:
http://cia.vc/stats/author/Pootle%20%3Cpootle@...>

I don't know if they allow third parties access to commit to version
control, but perhaps VMiklos can comment.

> B) keep RCS translation; throw away the local translation
> C) keep the local translation; turn the RCS translation into a suggestion
> D) keep the local translation; throw away the RCS translation
> 
> I would think, that similar conflicts do/will also arise in other situations
> as well:
> - two translators submitting a translation via the web interface within a short
> timeframe - for now the one who submits later is the "winner" and will never
> know about the translation, that was done before

This is not yet handled.

> - not yet existing: synchronization with other translation sources (offline
> translation editors, other translation servers (like launchpad),
> translations-by-mail, ...)
> 

You forgot one: file upload. This we do similarly: In the default upload
mode (merge) only untranslated entries receive the translations from the
uploaded file. In other cases the uploaded translation becomes a
suggestion that needs to be reviewed. Since 0.11 users with the relevant
right can choose to rather follow overwrite mode, where the file is
replaced in its entirety.

> I do not have a real plan yet, how this should be implemented. But I would be
> happy to work on it after some discussions (just in case, this is perceived as
> a to-be-solved problem by the developers).
> 

I'm sure there are more improvements that can be worked on, so we'll be
very happy to have more people interested in these functionalities.

> Anyway, I will move versioncontrol.py to a seperate directory and split it into
> different files for different revision control systems. Then it should be easy
> to move it to the toolkit, if necessary.
> 
Ok, I will commit the intermediate work anyway which might make it
easier in future to track the changes. (I might only get to it on
Monday, but I'll try to make it sooner in case it is important to
somebody.)

> > The only major shortcoming I currently see in the version
> > control functionality of Pootle is the ability to commit / update entire
> > directories/projects at a time. I don't know to what extent we would
> > have to extend the versioncontrol.py for that. 
> the current code interface should be sufficient for that: I will do this in the
> first week of July. But I am not sure where to integrate this into the web
> interface.
> 
> Lars

I didn't realise that we'll handle directories as well, but I see that
now. I'm just wondering now: is it ideal that we are recreating objects
all the time? I'm sure you implemented it with the functions to make the
diff in other files smaller (which I appreciate a lot :-)  but I guess
in future we might want to look at storing a version control object in
the project. Then again, currently it is possible to have different
version control systems for different languages in a project, for
example, which is not bad :-)

Just some notes. Thanks for your contributions.

Friedel

Hi,

> > For me, an "item" would be a single translated string (and possible meta
> > data). Right now the revision control interface does not care for indiv=
ual
> > items, but can only handle complete files. Thus any changes made through
> > the web interface are thrown away during an update from the version con=
trol
> > repository. The same goes for the other direction: intermediate changes=
 in
> > the version control repository are (theoretically) ignored and overwrit=
ten
> > by the changes made through the web interface when committing.
>=20
> There is already code to handle these situations. A version control
> conflict could cause an unparsable file, so we have to handle it for the
> users.

the current code does the following for different situations for conflicting
files (both local changes and changes in the RCS):
A) for commit: refuse to commit, since the local working copy is outdated
 (at least Subversion will handle it that way - I am not sure about the oth=
ers)
B) for update: revert (throw away) the local change and retrieve the changes
from RCS
 (valid for subversion/cvs/darcs)


> > In the end, this boils down to resolving conflicts, I guess.
> >=20
> > I could imagine, that the web interface should offer the following opti=
ons
> > for any conflicting item during revision control synchronization (updat=
e or
> > commit): A) keep RCS (revision control system) translation; turn the lo=
cal
> > translation into a suggestion
>=20
> This is the current behaviour. We always consider version control to be
> the authority.

not really - for now the "turn the local translation into a suggestion"
behaviour is only a feature request. In reality all local changes are lost.


> We might want to look at the detail to ensure that it is exactly what we
> want, but I know people are already using this feature. The darcs
> support comes from the people at Frugalware and their progress can be
> viewed here:
> http://cia.vc/stats/author/Pootle%20%3Cpootle@...>
>=20
> I don't know if they allow third parties access to commit to version
> control, but perhaps VMiklos can comment.

I guess, we (http://cryptobox.org) are maybe using it in a similar way: most
translations are done through the web interface and sometimes "third
parties" (developers creating translations locally) submit their changes via
svn. For now we did not run into problems, as there are only few translators
and developers.
The svn changes are updated hourly on the pootle server (by a simple custom=
 cron
script), Without this script there is (from my point of view) currently no
way to use both pootle and subversion as sources of translations.


> > - not yet existing: synchronization with other translation sources (off=
line
> > translation editors, other translation servers (like launchpad),
> > translations-by-mail, ...)
> >=20
>=20
> You forgot one: file upload.

oh - yes - that is quite important, too


> This we do similarly: In the default upload mode (merge) only untranslated
> entries receive the translations from the uploaded file. In other cases t=
he
> uploaded translation becomes a suggestion that needs to be reviewed.

great! I will take a look at the code soon.
=46rom my point of view, this merging code maybe should be used for all sourc=
es of
translations - including revision control system. The problem to be solved =
is
always the same - just the source of a translated string differs.


> > Anyway, I will move versioncontrol.py to a seperate directory and split=
 it
> > into different files for different revision control systems. Then it sh=
ould
> > be easy to move it to the toolkit, if necessary.
> >=20
> Ok, I will commit the intermediate work anyway which might make it
> easier in future to track the changes. (I might only get to it on
> Monday, but I'll try to make it sooner in case it is important to
> somebody.)

perfectly fine for me, as I will not start working on it before next Sunday.


> I didn't realise that we'll handle directories as well, but I see that
> now. I'm just wondering now: is it ideal that we are recreating objects
> all the time?

you mean, creating instances of the classes SVN, CVS and DARCS?
Does this create problems?


> I'm sure you implemented it with the functions to make the
> diff in other files smaller (which I appreciate a lot :-)  but I guess
> in future we might want to look at storing a version control object in
> the project.

ah! - you mean: storing the type of version control system as a property of=
 each
project?


> Then again, currently it is possible to have different version control
> systems for different languages in a project, for example, which is not
> bad :-)

another advantage (if I understand you right) would be, that no other piece=
 of
code needs to know about the different supported revision control systems. =
Thus
I think it is inevitable to have something similar to the current generic
wrapper functions.

On Sa, 2007-06-23 at 11:28 +0200, lars wrote:
> Hi,
> 
> > > For me, an "item" would be a single translated string (and possible meta
> > > data). Right now the revision control interface does not care for indivual
> > > items, but can only handle complete files. Thus any changes made through
> > > the web interface are thrown away during an update from the version control
> > > repository. The same goes for the other direction: intermediate changes in
> > > the version control repository are (theoretically) ignored and overwritten
> > > by the changes made through the web interface when committing.
> > 
> > There is already code to handle these situations. A version control
> > conflict could cause an unparsable file, so we have to handle it for the
> > users.
> 
> the current code does the following for different situations for conflicting
> files (both local changes and changes in the RCS):
> A) for commit: refuse to commit, since the local working copy is outdated
>  (at least Subversion will handle it that way - I am not sure about the others)
> B) for update: revert (throw away) the local change and retrieve the changes
> from RCS
>  (valid for subversion/cvs/darcs)

Are you sure the changes were not converted to suggestions? They can
seem gone if there were really big changes in the PO file (if #:
location comments also change as they can do in traditional gettext
based projects). Unfortunately we use those comments to identify
suggestions for PO based projects.

> 
> 
> > > In the end, this boils down to resolving conflicts, I guess.
> > > 
> > > I could imagine, that the web interface should offer the following options
> > > for any conflicting item during revision control synchronization (update or
> > > commit): A) keep RCS (revision control system) translation; turn the local
> > > translation into a suggestion
> > 
> > This is the current behaviour. We always consider version control to be
> > the authority.
> 
> not really - for now the "turn the local translation into a suggestion"
> behaviour is only a feature request. In reality all local changes are lost.
> 

Did you check the code in projects.py::updatepofile()?  Check where it
calls pootlefile.py::mergeitem()

> 
> > We might want to look at the detail to ensure that it is exactly what we
> > want, but I know people are already using this feature. The darcs
> > support comes from the people at Frugalware and their progress can be
> > viewed here:
> > http://cia.vc/stats/author/Pootle%20%3Cpootle@...>
> > 
> > I don't know if they allow third parties access to commit to version
> > control, but perhaps VMiklos can comment.
> 
> I guess, we (http://cryptobox.org) are maybe using it in a similar way: most
> translations are done through the web interface and sometimes "third
> parties" (developers creating translations locally) submit their changes via
> svn. For now we did not run into problems, as there are only few translators
> and developers.
> The svn changes are updated hourly on the pootle server (by a simple custom cron
> script), Without this script there is (from my point of view) currently no
> way to use both pootle and subversion as sources of translations.
> 

We have not used it extensively ourselves, but I'm quite sure that it
works. Perhaps if there is a real case that doesn't work, we should put
it in bugzilla and look at fixing it.

> 
> > > - not yet existing: synchronization with other translation sources (offline
> > > translation editors, other translation servers (like launchpad),
> > > translations-by-mail, ...)
> > > 
> > 
> > You forgot one: file upload.
> 
> oh - yes - that is quite important, too
> 
> 
> > This we do similarly: In the default upload mode (merge) only untranslated
> > entries receive the translations from the uploaded file. In other cases the
> > uploaded translation becomes a suggestion that needs to be reviewed.
> 
> great! I will take a look at the code soon.
> >From my point of view, this merging code maybe should be used for all sources of
> translations - including revision control system. The problem to be solved is
> always the same - just the source of a translated string differs.

I agree, it has been on my todo list for more than 6 months now. I would
really like something that merges this code as well as pomerge and
pot2po in the toolkit. They all have something in common. I'm just not
sure that it will be worthwhile or easy enough to really abstract
something useful out of all of them. Any improvement is good, though.
The code is a bit hard to read through.

> 
> 
> > > Anyway, I will move versioncontrol.py to a seperate directory and split it
> > > into different files for different revision control systems. Then it should
> > > be easy to move it to the toolkit, if necessary.
> > > 
> > Ok, I will commit the intermediate work anyway which might make it
> > easier in future to track the changes. (I might only get to it on
> > Monday, but I'll try to make it sooner in case it is important to
> > somebody.)
> 
> perfectly fine for me, as I will not start working on it before next Sunday.
> 
> 
> > I didn't realise that we'll handle directories as well, but I see that
> > now. I'm just wondering now: is it ideal that we are recreating objects
> > all the time?
> 
> you mean, creating instances of the classes SVN, CVS and DARCS?
> Does this create problems?

Not really. I just want to keep a check on the number of objects
created.

> 
> 
> > I'm sure you implemented it with the functions to make the
> > diff in other files smaller (which I appreciate a lot :-)  but I guess
> > in future we might want to look at storing a version control object in
> > the project.
> 
> ah! - you mean: storing the type of version control system as a property of each
> project?
> 
Yes, but the issue below is what would make me hesitant.

> 
> > Then again, currently it is possible to have different version control
> > systems for different languages in a project, for example, which is not
> > bad :-)
> 
> another advantage (if I understand you right) would be, that no other piece of
> code needs to know about the different supported revision control systems. Thus
> I think it is inevitable to have something similar to the current generic
> wrapper functions.
> 

Yes, the rest of the code should not care about that. Let us see where
we are after the initial refactoring and see where we want to go from
there.

F

Hi,

> > > This is the current behaviour. We always consider version control to be
> > > the authority.
> > 
> > not really - for now the "turn the local translation into a suggestion"
> > behaviour is only a feature request. In reality all local changes are lost.
> > 
> 
> Did you check the code in projects.py::updatepofile()?  Check where it
> calls pootlefile.py::mergeitem()

uh - you got me!
I was only looking at the code in versioncontrol.py and not in the places where
it is used. May fault - thanks for the pointer.
I never tried to use this feature, as it was mentioned as a feature request on
http://translate.sourceforge.net/wiki/pootle/version_control - or does the
update feature request involve something else? I was not sure, so maybe you
could update it to clarify the current state?


> > From my point of view, this merging code maybe should be used for all
> > sources of
> > translations - including revision control system. The problem to be solved
> > is always the same - just the source of a translated string differs.
> 
> I agree, it has been on my todo list for more than 6 months now. I would
> really like something that merges this code as well as pomerge and
> pot2po in the toolkit. They all have something in common. I'm just not
> sure that it will be worthwhile or easy enough to really abstract
> something useful out of all of them. Any improvement is good, though.
> The code is a bit hard to read through.

I just noticed :)
Sounds like something interesting - I will take a look at it.


> > > I didn't realise that we'll handle directories as well, but I see that
> > > now. I'm just wondering now: is it ideal that we are recreating objects
> > > all the time?
> > 
> > you mean, creating instances of the classes SVN, CVS and DARCS?
> > Does this create problems?
> 
> Not really. I just want to keep a check on the number of objects
> created.

"objects" in the sense of "instance"?
As these instances are created in a sequence (not all at the same time), there
should be no real resource problem. The time need to create an instance seems
to be neglectable from my point of view. But maybe I miss something?

Lars

On Sa, 2007-06-23 at 13:20 +0200, lars wrote:
> Hi,
> 
> > > > This is the current behaviour. We always consider version control to be
> > > > the authority.
> > > 
> > > not really - for now the "turn the local translation into a suggestion"
> > > behaviour is only a feature request. In reality all local changes are lost.
> > > 
> > 
> > Did you check the code in projects.py::updatepofile()?  Check where it
> > calls pootlefile.py::mergeitem()
> 
> uh - you got me!
> I was only looking at the code in versioncontrol.py and not in the places where
> it is used. May fault - thanks for the pointer.
> I never tried to use this feature, as it was mentioned as a feature request on
> http://translate.sourceforge.net/wiki/pootle/version_control - or does the
> update feature request involve something else? I was not sure, so maybe you
> could update it to clarify the current state?
> 
> 

It seems you added that request, and I never saw it. Well, see it as
that your request has been implemented (before you asked ;-)



...

> > > > I didn't realise that we'll handle directories as well, but I see that
> > > > now. I'm just wondering now: is it ideal that we are recreating objects
> > > > all the time?
> > > 
> > > you mean, creating instances of the classes SVN, CVS and DARCS?
> > > Does this create problems?
> > 
> > Not really. I just want to keep a check on the number of objects
> > created.
> 
> "objects" in the sense of "instance"?
> As these instances are created in a sequence (not all at the same time), there
> should be no real resource problem. The time need to create an instance seems
> to be neglectable from my point of view. But maybe I miss something?
> 
> Lars

I was more concerned about memory consumption than creation time. But I
guess we should measure it rather than speculate.

On Vr, 2007-06-22 at 14:37 +0200, VMiklos wrote:
> Na Tue, Jun 19, 2007 at 10:10:24AM +0200, F Wolff pisal(a):
> > I don't know if you are subscribed to the Pootle mailing list, so I'm
> > forwarding this mail to you.
> 
> thanks, i was not subscribed
> 
> > Lars redid the current implementation of
> > the version control to improve several things, but did not yet test it
> > with CVS or Darcs.  Will you be able to test this for us and comment on
> > it?
> 
> sure, i'll test it soon. originally i wanted to write you when i did the
> test, but i don't want you to think that i'm dead :)

No problem :-)  Let us know when you had the time. We'll continue on it
in the meanwhile.

> 
> > +  ## TODO: could any user of darcs explain this path?
> > +  MARKER_DIR = "../../../_darcs"
> 
> yes, though this variable name is quite misleading. when you record to
> darcs, it records (== commits) the difference between the pristine tree
> (under /_darcs/pristine - or _darcs/current for older versions) with the
> working directory (everything outside _darcs)
> 
> so we need to reach the _darcs dir from po/project/language. that will
> result in ../../../_darcs
> 
> VMiklos
> 

Thanks for the response.

Does this mean that we will hit some problems if we store a project with
a directory structure under po/project/language or we only store a
certain project on the server under darcs?

Na Mon, Jun 25, 2007 at 10:56:37AM +0200, F Wolff <friedel@...=
> pisal(a):
> > sure, i'll test it soon. originally i wanted to write you when i did the
> > test, but i don't want you to think that i'm dead :)
>=20
> No problem :-)  Let us know when you had the time. We'll continue on it
> in the meanwhile.

okay now i've applied the patch and it seems that pootle still works
fine here :)

VMiklos

--=20
developer of Frugalware Linux - http://frugalware.org

Hello,

Na Thu, Jun 28, 2007 at 01:30:54PM +0200, VMiklos <vmiklos@...> pisal(a):
> okay now i've applied the patch and it seems that pootle still works
> fine here :)

i'm really sorry about that, but i did not test it correctly - i simply
not restarted the pootle server.. how lame i am ;)

so of course it did not work perfectly, but only a few minor changes are
needed:

http://frugalware.org/~vmiklos/patches/pootle_darcs_fixes.diff

1) we currectly check for marker dir both in the project root and under
/po/project/language. the old version just checked under the project
root. this is an issue with darcs, because we have to use the
../../../_darcs or the _darcs string for MARKER_DIR but we can't use
both. the first hunk fixes this - i think it does not break cvs or svn,
as they have a MARKER_DIR in each directory

2) the second hunk then updates the MARKER_DIR for darcs

3) the third hunk no longer converts the file's path to absolute.
probably i copy&pasted this from cvs and/or svn, but newer darcs
versions seem to no longer handle absolute paths correctly, while older
versions are working with relative paths, too (we always call darcs from
the project root so this is not a big issue)

sorry for being so slow,

- VMiklos

ps: if this looks good, i plan to send a patch to add git support :)

Hi VMiklos,

thanks for testing the changes!

> so of course it did not work perfectly, but only a few minor changes are
> needed:
> 
> http://frugalware.org/~vmiklos/patches/pootle_darcs_fixes.diff
> 
> 1) we currectly check for marker dir both in the project root and under
> /po/project/language. the old version just checked under the project
> root. this is an issue with darcs, because we have to use the
> ../../../_darcs or the _darcs string for MARKER_DIR but we can't use
> both. the first hunk fixes this

Does that mean, that the "_darcs" directory can be at any directory level above
(and including) the directory of the po file?
Then maybe it would be better just to walk all directories up to find it.

I attached a proposal for this - please test it. But maybe I just went into
the wrong direction, as I never used darcs.


> i think it does not break cvs or svn, as they have a MARKER_DIR in each
> directory

I am not sure about this: if I am not wrong, then the current directory is not
predictable during "_init_". Thus we may not assume, that the current directory
contains a svn/cvs directory?


> 2) the second hunk then updates the MARKER_DIR for darcs

see above


> 3) the third hunk no longer converts the file's path to absolute.
> probably i copy&pasted this from cvs and/or svn, but newer darcs
> versions seem to no longer handle absolute paths correctly, while older
> versions are working with relative paths, too (we always call darcs from
> the project root so this is not a big issue)

Are you sure, that we always call darcs from the project root? I do not see
this in the code. Did I miss something?

I am in no way familiar with darcs, but maybe we would need to add '--repodir'
to all 'darcs' calls? Maybe you could describe, how the arguments are treated,
when you use a --repodir? Relative to the current directory or relative to the
repodir?

regards,
Lars

PS: the currently attached patch is a little bit clumsy. I would improve it
later, but first I would like to know, if it solves any problems :)

Hello,

Na Fri, Jul 13, 2007 at 07:08:39PM +0200, lars <lists@...> pisal(=
a):
> Does that mean, that the "_darcs" directory can be at any directory level=
 above
> (and including) the directory of the po file?
> Then maybe it would be better just to walk all directories up to find it.

that's not a bad idea, but it was not supported even before the oo
rewrite and my patch was a fix, not an improvement :)

> I attached a proposal for this - please test it. But maybe I just went in=
to
> the wrong direction, as I never used darcs.

2007-07-13 20:25:24: Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/Pootle/users.py", line 235, in han=
dle
    page =3D self.getpage(pathwords, session, argdict)
  File "/usr/lib/python2.5/site-packages/Pootle/pootle.py", line 429, in ge=
tpage
    return indexpage.ProjectIndex(project, session, argdict)
  File "/usr/lib/python2.5/site-packages/Pootle/indexpage.py", line 353, in=
 __init__
    childitems =3D self.getchilditems(dirfilter)
  File "/usr/lib/python2.5/site-packages/Pootle/indexpage.py", line 581, in=
 getchilditems
    fileitem =3D self.getfileitem(childfile)
  File "/usr/lib/python2.5/site-packages/Pootle/indexpage.py", line 752, in=
 getfileitem
    if versioncontrol.hasversioning(os.path.join(self.project.podir, self.d=
irname)):
  File "/usr/lib/python2.5/site-packages/Pootle/versioncontrol.py", line 35=
8, in hasversioning
    get_versioned_object(item)
  File "/usr/lib/python2.5/site-packages/Pootle/versioncontrol.py", line 33=
4, in get_versioned_object
    return vers_sys(location)
  File "/usr/lib/python2.5/site-packages/Pootle/versioncontrol.py", line 28=
4, in __init__
    self.location =3D realpath_pofile.replace(basedir, 1)
TypeError: coercing to Unicode: need string or buffer, int found

> > i think it does not break cvs or svn, as they have a MARKER_DIR in each
> > directory
>=20
> I am not sure about this: if I am not wrong, then the current directory i=
s not
> predictable during "_init_". Thus we may not assume, that the current dir=
ectory
> contains a svn/cvs directory?

umm, just try started PootleServer from /root and the os.getcwd() in
GenericVersionControlSystem's __init__() says
/usr/lib/python2.5/site-packages/Pootle, so i think we can assume

> > 3) the third hunk no longer converts the file's path to absolute.
> > probably i copy&pasted this from cvs and/or svn, but newer darcs
> > versions seem to no longer handle absolute paths correctly, while older
> > versions are working with relative paths, too (we always call darcs from
> > the project root so this is not a big issue)
>=20
> Are you sure, that we always call darcs from the project root? I do not s=
ee
> this in the code. Did I miss something?

i just added a few "print os.getcwd()" to debug the code ans see that we
do so, at least now i tried to several actions and i was not able to
call darcs not from the project root

> I am in no way familiar with darcs, but maybe we would need to add '--rep=
odir'
> to all 'darcs' calls? Maybe you could describe, how the arguments are tre=
ated,
> when you use a --repodir? Relative to the current directory or relative t=
o the
> repodir?

i never used --repodir, but as far as i see it's relative to the repodir

> PS: the currently attached patch is a little bit clumsy. I would improve =
it
> later, but first I would like to know, if it solves any problems :)

sure, that's not a problem :)

i haven't checked deeply what's the problem with it because i don't see
exactly what's the problem with my patch which works here :)

two notes:

1) i know that i assume something that is not explicitly coded (we call
darcs from the pootle root), but i think it's still true

2) currently an scm is enabled if the marked dir is found in the
/po/project/lang dir _and_ in the / dir, while my patch checks only the
/ dir. this obviously fixes the darcs behaviour while i'm not sure how
this can cause problems with cvs/svn :)

once it works again, then of course we could improve it so that it could
find the _darcs dir not only in the pootle root

thanks,
- VMiklos

Hi,

> > Then maybe it would be better just to walk all directories up to find it.
> 
> that's not a bad idea, but it was not supported even before the oo
> rewrite and my patch was a fix, not an improvement :)
sorry - I did not mean to offend you ...

> "/usr/lib/python2.5/site-packages/Pootle/versioncontrol.py", line 284, in
> __init__ self.location = realpath_pofile.replace(basedir, 1) TypeError:
> coercing to Unicode: need string or buffer, int found
sorry again for not really testing it - the line should be:
  self.location = realpath_pofile.replace(basedir, "", 1)
I attached a fixed patch.

> > > i think it does not break cvs or svn, as they have a MARKER_DIR in each
> > > directory
> > [...]
> umm, just try started PootleServer from /root and the os.getcwd() in
> GenericVersionControlSystem's __init__() says
> /usr/lib/python2.5/site-packages/Pootle, so i think we can assume

For my setup it constantly says "/etc/pootle".
My pootle projects are below /var/lib/pootle/ (debian).
Additionally for my setup the root directory of the projects is not controlled
by svn, thus there is no MARKER_DIR at this level.


> > I am in no way familiar with darcs, but maybe we would need to add
> > '--repodir' to all 'darcs' calls? Maybe you could describe, how the
> > arguments are treated, when you use a --repodir? Relative to the current
> > directory or relative to the repodir?
> 
> i never used --repodir, but as far as i see it's relative to the repodir
ok - thanks for the information


> 2) currently an scm is enabled if the marked dir is found in the
> /po/project/lang dir _and_ in the / dir, while my patch checks only the
> / dir. this obviously fixes the darcs behaviour while i'm not sure how
> this can cause problems with cvs/svn :)
As only a part of my project's root is controlled by svn, the scm directory is
not detected anymore with the patch.
I think, the same problem could occour for darcs users with a similar setup,
too. Or do I still miss the point regarding darcs? (I only know my way around
with svn)

Could you please check the fixed patch again?

regards,
Lars

Hello,

Na Fri, Jul 13, 2007 at 11:45:42PM +0200, lars <lists@...> pisal(=
a):
> > > Then maybe it would be better just to walk all directories up to find=
 it.
> >=20
> > that's not a bad idea, but it was not supported even before the oo
> > rewrite and my patch was a fix, not an improvement :)
> sorry - I did not mean to offend you ...

i just wanted to say that maybe my mail was too offending ;)

> > "/usr/lib/python2.5/site-packages/Pootle/versioncontrol.py", line 284, =
in
> > __init__ self.location =3D realpath_pofile.replace(basedir, 1) TypeErro=
r:
> > coercing to Unicode: need string or buffer, int found
> sorry again for not really testing it - the line should be:
>   self.location =3D realpath_pofile.replace(basedir, "", 1)
> I attached a fixed patch.

that works almost perfectly :)

there is a typo: it uses pathnames like Pootle/po/project/lang instead
of po/project/lang if the _darcs dir is under Pootle

a quick fix:=20
--- versioncontrol.py.lars	2007-07-13 23:55:12.000000000 +0200
+++ versioncontrol.py	2007-07-14 00:02:53.000000000 +0200
@@ -278,7 +278,7 @@
                 self.darcsdir =3D current_dir
                 # retrieve the relative path of the po file based on self.=
darcsdir
                 realpath_pofile =3D os.path.realpath(location)
-                basedir =3D os.path.dirname(self.darcsdir) + os.path.sep
+                basedir =3D self.darcsdir + os.path.sep
                 if realpath_pofile.startswith(basedir):
                     # remove the base directory (including the trailing sl=
ash)
                     self.location =3D realpath_pofile.replace(basedir, "",=
 1)

> For my setup it constantly says "/etc/pootle".
> My pootle projects are below /var/lib/pootle/ (debian).
> Additionally for my setup the root directory of the projects is not contr=
olled
> by svn, thus there is no MARKER_DIR at this level.

aha. so then the problem at me is that the .prefs files are in the same
dir as the projects, hence i haven't noticed the problem

> As only a part of my project's root is controlled by svn, the scm directo=
ry is
> not detected anymore with the patch.
> I think, the same problem could occour for darcs users with a similar set=
up,
> too. Or do I still miss the point regarding darcs? (I only know my way ar=
ound
> with svn)

no, you have reason, i haven't though of the ability to have the config
files and the projects under different dirs

thanks,
- VMiklos

Hi,

> there is a typo: it uses pathnames like Pootle/po/project/lang instead
> of po/project/lang if the _darcs dir is under Pootle
hehe - that looks more like a brain-bug of mine, than a 'typo' ;)

fine - it feels, like we finally found the perfect patch: it is attached for
someone with svn write access ...

thanks for thoroughly checking and fixing it!

reagards,
Lars

Op Saterdag 14-07-2007 om 00:41 uur [tijdzone +0200], schreef lars:
> Hi,
> 
> > there is a typo: it uses pathnames like Pootle/po/project/lang instead
> > of po/project/lang if the _darcs dir is under Pootle
> hehe - that looks more like a brain-bug of mine, than a 'typo' ;)
> 
> fine - it feels, like we finally found the perfect patch: it is attached for
> someone with svn write access ...
> 
> thanks for thoroughly checking and fixing it!
> 
> reagards,
> Lars

Thanks for all the work! I especially appreciate the time put into
comments. This is committed in revision 6117. Shall we now consider
splitting this into different files as we move it to the toolkit?

F

Hello,

Na Mon, Jul 16, 2007 at 03:51:02PM +0200, F Wolff <friedel@...> pisal(a):
> Thanks for all the work! I especially appreciate the time put into
> comments. This is committed in revision 6117. Shall we now consider
> splitting this into different files as we move it to the toolkit?

i already have a patch here for git support, should i post it or first
move it to the toolkit and post after it?

- VMiklos

Op Maandag 16-07-2007 om 16:07 uur [tijdzone +0200], schreef VMiklos:
> Hello,
> 
> Na Mon, Jul 16, 2007 at 03:51:02PM +0200, F Wolff <friedel@...> pisal(a):
> > Thanks for all the work! I especially appreciate the time put into
> > comments. This is committed in revision 6117. Shall we now consider
> > splitting this into different files as we move it to the toolkit?
> 
> i already have a patch here for git support, should i post it or first
> move it to the toolkit and post after it?
> 
> - VMiklos

It doesn't really matter to me. The split might rather affect your patch
than the move. Where in the toolkit should we move to?
translate/storage/versioncontrol/ ?

Op Saterdag 14-07-2007 om 00:41 uur [tijdzone +0200], schreef lars:
> +            while not os.path.isdir(os.path.join(current_dir,
> self.MARKER_DIR)):
> +                if os.path.samefile(current_dir, \
> +                        os.path.join(current_dir, os.path.pardir)):
> +                    # we reached the root directory - stop
> +                    break 

os.path.samefile() is not available on Windows. Can we consider an
alternative solution? Perhaps by comparing absolute path names?

F

Hi,

> > +            while not os.path.isdir(os.path.join(current_dir,
> > self.MARKER_DIR)):
> > +                if os.path.samefile(current_dir, \
> > +                        os.path.join(current_dir, os.path.pardir)):
> > +                    # we reached the root directory - stop
> > +                    break 
> 
> os.path.samefile() is not available on Windows. Can we consider an
> alternative solution? Perhaps by comparing absolute path names?

I didn't check the availability of os.path.samefile - sorry!

I attached an alternative condition.
To make sure, that everything works as expected, maybe someone with access to a
windows environment could check, if the following condition returns true?
 os.path.dirname('E:\') == 'E:\'
I cannot test it here ...

regards,
Lars

Op Dinsdag 17-07-2007 om 17:39 uur [tijdzone +0200], schreef lars:

...

> I didn't check the availability of os.path.samefile - sorry!

I ran the unit tests under wine and spotted the problem :-)

> 
> I attached an alternative condition.
> To make sure, that everything works as expected, maybe someone with access to a
> windows environment could check, if the following condition returns true?
>  os.path.dirname('E:\') == 'E:\'
> I cannot test it here ...
> 
> regards,
> Lars
> 

I don't have windows, but Wine seems happy. I'll wait another day for
somebody to test on Windows, but I believe this is ok.

Thanks for the quick response.

F