Docutils: Documentation Utilities

Felix Wiemann wrote:
> BerliOS works now (mostly; From headers of checkin mails won't contain
> the real name for now).
> 
> 
> This is how to test it:
> 
> ViewCVS is at <http://svn.berlios.de/viewcvs/docutils/>.
> 
Works nicely.

> To check out the sources, type:
> 
>   # anonymous
>   svn co svn://svn.berlios.de/docutils/trunk docutils

Works nicely.

>   # authenticated (please mail me your BerliOS username to get access)
>   svn co svn+ssh://username@...> docutils
> 
sfcben

> 
> This is what I propose, as a rough outline:
> 
> * Move the main tree and any sandboxes whose users want to migrate to
>   SVN to a fresh SVN-repository on BerliOS.
> 
I think all sandboxes should be copied over.
Anyboding having a sandbox probably also works with the main tree (at least to 
update) and so would have to use svn anyway.  Using svn for tree while 
sticking to cvs is awkward for work and awkward for moving code between 
sandbox and the tree (which happens periodically).
Anyway, I agree to moving sandbox/cben.

> * Create a support request on SF.net to delete the main tree and the
>   migrated sandboxes.
> 
Naturally we should wait with this until we have worked with BerliOS for some 
time and seen it's stable.

> * For each sandbox which wants to move later, convert it separately and
>   dump it into repository.  The revision dates in the sandbox history
>   will be lost, even though they are stored in a separate revision
>   property for completeness.  (I've tested this before.)
> 
> * For every batch of sandboxes that have been migrated, create a support
>   request on SF.net to delete them.
> 
Unecessary hassle as I've opined above.

> Do you think it's feasible?  Do you see any potential problems?
> 
> 
> Things to be done before migrating is possible:
> 
> * Create a version of the docutils-update script which fetches files
>   from an SVN repository (svn is installed on shell.sf.net).
> 
> * Create a document describing how to configure the client, how to check
>   out sources, how the repository is organized, etc.
> 
> * Write a repository-backup script.
> 
 From BerliOS's `Subversion (SVN) HowTo`__:

     [Top] How to get your SVN tree back [Next]

     Daily dumps of entire project SVN Repositories are available for backup or
     mirroring purposes. These are available at:
     http://svn.berlios.de/svndumps/projectname-repos.gz. Where projectname is
     the Unix name of your project.

__ https://developer.berlios.de/docman/display_doc.php?docid=394&group_id=2

It's a heavy one - 27M!  I'd say download it once or twice a week.  Where do 
we want to backup it?  Onto shell.sf.net?

I see they give direct repository access for project admins on svn.berlios.de. 
  That's really cool, it means we won't need to place support requests for 
anything SVN-related.  Goodbye SourceForge :-).

> Anything else?
> 
* People that have pending uncommitted changes in their local copies (e.g. I
   have a few) will be inconvenienced in the trasition.  This is actually good
   as it will force people to finish up their changes and commit ;-).  We will
   want a commit freeze anyway.

* After the conversion, it's good idea to see whether any files need fixing of
   ``svn:keywords``, ``svn:mime-type`` or ``svn:eol-style`` properties

* If we get a subversion repo, it sounds very tempting to install Trac on top
   of it.  After all it's a reST-based wiki which sounds like a perfect addon
   for docutils development.  As far as I understad from TracInstall, it wants
   to run on a machine that can locally access the repository, e.g.
   svn.berlios.de.  I don't think svn.berlios.de allows custom CGI?  I suppose
   it depends on BerliOS installing Trac themselves.

-- 
Beni Cherniavsky <cben@...>, who can only read email on weekends.

Beni Cherniavsky wrote:

> Felix Wiemann wrote:
>
>>   # authenticated (please mail me your BerliOS username to get access)
>
> sfcben

Added.

>> This is what I propose, as a rough outline:
>>
>> * Move the main tree and any sandboxes whose users want to migrate to
>>   SVN to a fresh SVN-repository on BerliOS.
>
> I think all sandboxes should be copied over.

I don't want to force anybody to switch to Subversion.

> Anyboding having a sandbox probably also works with the main tree (at
> least to update) and so would have to use svn anyway.

I'll probably send mails to all people who have a sandbox.  But I'm not
sure yet what to do in cases where I don't get a reply at all.  Maybe
just move these sandboxes?

>> * Create a support request on SF.net to delete the main tree and the
>>   migrated sandboxes.

I'm just seeing that they want to keep the data:
<http://sourceforge.net/project/admin/removal.php?group_id=38414#project_relocation>.

>> * Write a repository-backup script.
>
> From BerliOS's `Subversion (SVN) HowTo`__:
>
>      Daily dumps of entire project SVN Repositories are available for backup or
>      mirroring purposes.
>
> It's a heavy one - 27M!

That's too much.  I think it's better to copy just new revisions, which
should be quite easy with FSFS.

> I'd say download it once or twice a week.

It should be possible to do it daily, given that the amount of data to
transfer is very low.

> Where do we want to backup it?  Onto shell.sf.net?

On my local system (and anyone else's who volunteers to do backups).
shell.sf.net has too limited resources.

> I see they give direct repository access for project admins on
> svn.berlios.de.

For everyone.  That's why I think it's very important to do regular
backups.

> * People that have pending uncommitted changes in their local copies (e.g. I
>    have a few) will be inconvenienced in the trasition.  This is actually good
>    as it will force people to finish up their changes and commit ;-).

Or store a patch, or copy the CVS working copy over the SVN working
copy.

>    We will want a commit freeze anyway.

Yes (for a day or so, probably).

> * After the conversion, it's good idea to see whether any files need fixing of
>    ``svn:keywords``, ``svn:mime-type`` or ``svn:eol-style`` properties

Should be OK (automatically set by cvs2svn).

> * If we get a subversion repo, it sounds very tempting to install Trac on top
>    of it.  After all it's a reST-based wiki which sounds like a perfect addon
>    for docutils development.

I'm not sure if it's possible and if we need that.  Trac would have to
be installed and a Wiki needs to be maintained; I'm happy with the
mailing lists.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Executive summary: I'm +0 on moving to Subversion, as long as Felix
(or somebody else) takes responsibility to make sure it moves
smoothly.  I'm happy with SourceForge -- they provide a great service
for free -- and I'm not inconvenienced enough by the shortcomings of
CVS to bother with the SVN conversion.

I have very little time right now to test BerliOS.  I'll try to make
an account soon.

[Felix Wiemann]
 > This is what I propose, as a rough outline:
 >
 > * Move the main tree and any sandboxes whose users want to migrate
 >   to SVN to a fresh SVN-repository on BerliOS.

If we're going to move, let's go whole-hog.  Copy everything over:
core, all sandboxes, web, everything.

 > I don't want to force anybody to switch to Subversion.

The only way not to force anybody to switch would be to maintain the
current CVS repo in parallel.

The thing to do is to post the plan on docutils-develop (as you've
done ;-) and docutils-users, and get rough consensus.  Unless there
are significant and valid objections, just go ahead and do it.

 > * Create a support request on SF.net to delete the main tree and the
 >   migrated sandboxes.

I don't think we should delete anything until BerliOS (or wherever)
has fully proven itself.  Disable SF's CVS if possible, but leave the
data alone for now.

And as you later point out, SF wants to keep the data intact.  That's
fine by me, and better for the project IMO.

 > * For each sandbox which wants to move later, convert it separately
 >   and dump it into repository.  The revision dates in the sandbox
 >   history will be lost, even though they are stored in a separate
 >   revision property for completeness.  (I've tested this before.)

Are all revision dates lost, or only sandbox files?  Are they special
because of the separate conversion?  If so, more reason to convert
everything all at once.

 > Do you think it's feasible?  Do you see any potential problems?

I think most developers will adapt quickly and easily.  As long as we
still provide snapshots (which could be hosted on BerliOS) for
download, I don't see any technical problem.

The biggest issue for me is reliability.  SourceForge has a proven
track record.  But as long as we make regular repository backups, the
project will survive almost any calamity unscathed (mostly).

 > Things to be done before migrating is possible:
 >
 > * Create a version of the docutils-update script which fetches files
 >   from an SVN repository (svn is installed on shell.sf.net).

The SF pages could simply forward to some other site, if that's
easier.

 > * Create a document describing how to configure the client, how to
 >   check out sources, how the repository is organized, etc.

There should be plenty of such info on the 'net.

 > * Write a repository-backup script.

The bulk of that work has probably already been done.

 > Anything else?

What would the repository layout be?  How would the current three
modules (docutils, web, sandbox) located in the standard SVN layout
(trunk, branches, tags)?  Please provide a layout plan.

--
David Goodger <http://python.net/~goodger>

David Goodger wrote:

> Felix Wiemann wrote:
>
>> This is what I propose, as a rough outline:
>>
>> * Move the main tree and any sandboxes whose users want to migrate
>>   to SVN to a fresh SVN-repository on BerliOS.
>
> If we're going to move, let's go whole-hog.  Copy everything over:
> core, all sandboxes, web, everything.

OK.

> The thing to do is to post the plan on docutils-develop (as you've
> done ;-) and docutils-users, and get rough consensus.  Unless there
> are significant and valid objections, just go ahead and do it.

OK.

>> * For each sandbox which wants to move later, convert it separately
>>   and dump it into repository.  The revision dates in the sandbox
>>   history will be lost, even though they are stored in a separate
>>   revision property for completeness.  (I've tested this before.)
>
> Are all revision dates lost, or only sandbox files?

Only for the files of the migrated sandbox.

> Are they special because of the separate conversion?

Yes.  Revisions must be ordered chronologically, because retrieving
revisions by date is done using a binary search.  A sandboxes which is
migrated later would add new revisions with old dates, so the chronology
would be broken.

> If so, more reason to convert everything all at once.

OK.

>> Things to be done before migrating is possible:
>>
>> * Create a version of the docutils-update script which fetches files
>>   from an SVN repository (svn is installed on shell.sf.net).
>
> The SF pages could simply forward to some other site, if that's
> easier.

It doesn't make much of a difference, probably.  I'm happy with the site
staying on SF.net.

>> * Create a document describing how to configure the client, how to
>>   check out sources, how the repository is organized, etc.
>
> There should be plenty of such info on the 'net.

I'll have a look.  Maybe not enough, or not specific enough.

>> * Write a repository-backup script.
>
> The bulk of that work has probably already been done.

Yup, I'll google a bit.

> What would the repository layout be?  How would the current three
> modules (docutils, web, sandbox) located in the standard SVN layout
> (trunk, branches, tags)?  Please provide a layout plan.

If the "docutils/" directory is at
svn+ssh://svn.berlios.de/svnroot/repos/docutils, then it looks like
this:

docutils/
|-- branches/
|   |-- branch1/
|   |   |-- docutils/
|   |   |-- sandbox/
|   |   `-- web/
|   `-- branch2/
|       |-- docutils/
|       |-- sandbox/
|       `-- web/
|-- tags/
|   |-- tag1/
|   |   |-- docutils/
|   |   |-- sandbox/
|   |   `-- web/
|   `-- tag2/
|       |-- docutils/
|       |-- sandbox/
|       `-- web/
`-- trunk/
    |-- docutils/
    |   |-- BUGS.txt
    |   |-- COPYING.txt
    |   `-- ...
    |-- sandbox/
    |   |-- aahz
    |   |-- agurtovoy
    |   |-- amk
    |   |-- ax-
    |   |-- bbum
    |   |-- blais
    |   |-- cben
    |   |-- davidg
    |   |-- delza
    |   |-- dkuhlman
    |   `-- ...
    `-- web/
        |-- PyBanner016.png
        |-- default.css
        |-- index.txt
        `-- ...

Quite typical layout, actually.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

The repository layout looks good to me.

I've registered at BerliOS, user "goodger".

--
David Goodger <http://python.net/~goodger>

David Goodger wrote:

> As long as we still provide snapshots (which could be hosted on
> BerliOS) for download, I don't see any technical problem.

Oh, by the way, maybe docutils-update needs to be split, because the
snapshot should probably live on the anonymous FTP at BerliOS.  (SF
doesn't seem to like snapshot files served via the website.)

See
<http://developer.berlios.de/docman/display_doc.php?docid=57&group_id=2>.
(FTP isn't currently activated for docutils.)

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

David Goodger wrote:

> If we're going to move, let's go whole-hog.  Copy everything over:
> core, all sandboxes, web, everything.

I'll upload a new repository containing all sandboxes in a few minutes.

Note that you need to create new working copies because the revision
numbers have changed.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

(Posted to Docutils-develop and Docutils-users.  Please reply to only
one mailing list.)

I would like to move the complete Docutils CVS repository (including web
site files and sandboxes) to Subversion_ (SVN) within the next days.

.. _Subversion: http://subversion.tigris.org/

The Subversion server is at svn.berlios.de; see
<https://developer.berlios.de/projects/docutils/> for the project page.

I wrote up a preliminary document about how to access the Subversion
repository; see
<http://docutils.sourceforge.net/docs/dev/repository.html>.  The SVN
repository is currently accessible, but working copies will become
invalid when I upload a new (updated) repository in a few days.

The repository and the snapshot files will live at BerliOS.  The web
site and all other project-related things (bug tracker etc.) will stay
at SourceForge.

If you already have developer (write) access on SF.net, please just mail
me your BerliOS user name and I'll add you to the Docutils project on
BerliOS, so that you can read-write access the Subversion repository.

Do you have any questions or concerns regarding the move?

If there are no unresolved issues left, I'll announce a checkin freeze
in a few days and move the most recent repository tree to BerliOS; then
repository access will be possible via Subversion.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Felix Wiemann wrote:

> If you already have developer (write) access on SF.net, please just mail
> me your BerliOS user name and I'll add you to the Docutils project on
> BerliOS, so that you can read-write access the Subversion repository.

The BerliOS web server was down for some hours; registration should be
possible again.  The BerliOS website is at
<http://developer.berlios.de/>, you can register at
<https://developer.berlios.de/account/register.php>.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

>>>>> "Felix" =3D=3D Felix Wiemann <Felix.Wiemann@...> writes:

    Felix> * Create a version of the docutils-update script which
    Felix> fetches files from an SVN repository (svn is installed on
    Felix> shell.sf.net).

Please have a look at `rstpc.py`_ that implements a very similar goal
for the DocIt site. Given a slightly different goals, it has the
ability of reading dependencies and build a different document that
includes the changed one.

Beware, it's far from perfect [*]_, as everything, svn included, is :)

ciao, lele.

PS: I just started my 'lele' account on BerliOS :-)

.. _rstpc.py: http://docit.bice.dyndns.org/static/www/src/rstpc.py

.. [*] now and then, doing the svn update, svn leaves locks around,
       although this may be very well due to my usage of python
       bindings.
--=20
nickname: Lele Gaifax	| Quando vivr=C3=B2 di quello che ho pensato ieri
real: Emanuele Gaifas	| comincer=C3=B2 ad aver paura di chi mi copia.
email: lele@...	|		-- Fortunato Depero, 1929.

Lele Gaifax wrote:

> Felix Wiemann writes:
>
>> * Create a version of the docutils-update script which fetches files
>>   from an SVN repository (svn is installed on shell.sf.net).
>
> Please have a look at `rstpc.py`_ that implements a very similar goal
> for the DocIt site.

Apart from the fact that I've just written a halfway-working SVN version
of docutils-update, I don't want to move the Docutils web site away from
SF.net, so it isn't easily possible to implement updating the HTML files
from a post-commit hook.  So I think it's OK to stay with
docutils-update for now.  Thanks nonetheless!

> PS: I just started my 'lele' account on BerliOS :-)

Added; thanks.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Felix Wiemann wrote:

> If there are no unresolved issues left, I'll announce a checkin freeze
> in a few days and move the most recent repository tree to BerliOS; then
> repository access will be possible via Subversion.

OK, here we go:

**Please stop committing changes to the repository now.**

As soon as the latest revisions turn up on the nightly CVS repository
snapshot, I'll convert the repository, upload it to BerliOS, and
announce it here.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

[Felix Wiemann]
 > OK, here we go:
 >
 > **Please stop committing changes to the repository now.**

Now is not a good time.  PyConDC has begun, and I am doing some
Docutils development.  I'd prefer if you start the freeze on Saturday
(after PyCon) or next week.

--
David Goodger <http://python.net/~goodger>

[stripped Docutils-users from CC, re-add if necessary]

David Goodger wrote:

> Felix Wiemann wrote:
>
>> **Please stop committing changes to the repository now.**
>
> Now is not a good time.  PyConDC has begun, and I am doing some
> Docutils development.

Mmmh.  I think I can complete it within 2-3 hours.  Is that a problem,
should I delay it nonetheless?

i-wish-i-were-at-pycon-too-ly y'rs, Felix

-- 
Felix Wiemann -- http://www.ososo.de/

Felix Wiemann wrote:

> Mmmh.  I think I can complete it within 2-3 hours.

That is because I expect the next snapshot to turn up in about 60
minutes.

(You could even check in something before, if you're quick enough ;).)

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Felix Wiemann wrote:

> Felix Wiemann wrote:
>
>> Mmmh.  I think I can complete it within 2-3 hours.
>
> That is because I expect the next snapshot to turn up in about 60
> minutes.

I was wrong.  The nightly repository snapshot don't seem to be created
every 24 hours.

So please go ahead checkin in changes again.  I'll try again next
Saturday.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Felix Wiemann wrote:

> So please go ahead checkin in changes again.  I'll try again next
> Saturday.

The new repository snapshot finally arrived.  :)

Since there were no intermediate check-ins to the CVS repository, I've
been able to move the repository to BerliOS (thanks, David, for your
patience).  Oh, and sorry for making so much noise on the mailing lists
before.

See <http://docutils.sourceforge.net/docs/dev/repository.html> for how
to access the repository.  If you are a member of the Docutils project
at SourceForge, please register at BerliOS
(https://developer.berlios.de/account/register.php) and mail me your
BerliOS user name.

If you encounter any problems, please feel free to send me an email or
post it on the mailing list.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Felix Wiemann wrote:

> [...] I've been able to move the repository to BerliOS [...]

... and this is what I forgot to mention:

* Please don't commit changes to the old CVS repository anymore; the
  Subversion repository should be used exclusively to avoid confusion.

* If you have a working copy of the test repository I had set up, please
  delete it and get a fresh checkout.  The new repository is different,
  and svn would be confused.

-- 
When replying to my email address, please ensure
that the mail header contains 'Felix Wiemann'.

http://www.ososo.de/

Felix Wiemann wrote:
> 
> Mark Nodine wrote:
> 
> > I'm having absolutely no luck trying to connect to shell.berlios.de
> > through our firewall, either by ssh or telnet.  Since I can't ssh, I
> > can't get the svn+ssh: URL to work, either.
>
> BerliOS also offers CVS, but I presume you'd have the same problem since
> developer CVS access is done via SSH as well.

Yes, it would have the same problem.
 
> I'm not sure what's the best solution, but these are my ideas (not all
> of them may be viable for you):
> 
> * Ask your admin to open the SSH port.

Requested and emphatically denied, since you can use ssh to
create an "unauthenticated tunnel" through the firewall.

> > Is there an http: equivalent for the repository?
> 
> No, unfortunately not; BerliOS seems to be ignoring requests for WebDAV
> support.

Is the problem that they are not running an Apache server?
Would the existence proof of a would-be contributor who is
unable to do so without WebDAV access help to turn up the
heat?

> * Find an SVN hoster who allows access via HTTP.
> * Host the repository on your company's server, making the repository
>   anonymously accessible, or frequently making public releases (e.g. on
>   SF.net).
> * Use an alternative version control system which doesn't require SSH
>   access (GNU arch?).

Would there be any way to convince SourceForge to expand their
offerings to Subversion?  That'd seem a natural place for hosting
the docutils repository if they would.

	--Mark

Mark Nodine wrote:
> Felix Wiemann wrote:
>>* Ask your admin to open the SSH port.
> 
> Requested and emphatically denied, since you can use ssh to
> create an "unauthenticated tunnel" through the firewall.

as long as you have any outgoing connection there is the possibility to 
create a "unauthenticated tunnel". guess what httptunnel is for ;-)

> Would there be any way to convince SourceForge to expand their
> offerings to Subversion?

well with one million users, one vote is not much. on the other hand, i 
guess there are others that like to have svn too.

chris

[Mark Nodine]
> Felix Wiemann wrote:
>> * Ask your admin to open the SSH port.
>
> Requested and emphatically denied, since you can use ssh to
> create an "unauthenticated tunnel" through the firewall.

How about asking your admin to open a dedicated port on your end that
connects to svn+ssh://svn.berlios.de only?

>>> Is there an http: equivalent for the repository?
>>
>> No, unfortunately not; BerliOS seems to be ignoring requests for
>> WebDAV support.
>
> Is the problem that they are not running an Apache server?
> Would the existence proof of a would-be contributor who is
> unable to do so without WebDAV access help to turn up the
> heat?

I don't know.  Try telling them, and see?

> Would there be any way to convince SourceForge to expand their
> offerings to Subversion?

Here's their policy as of 2004-10-20:
http://sourceforge.net/tracker/index.php?func=detail&aid=1050616&group_id=1&atid=200001
I don't see how it would help you though.  You'd still be in the same
position, behind a firewall closed to SSH.

> That'd seem a natural place for hosting
> the docutils repository if they would.

Yes, that would have been ideal.  But it doesn't exist now and
doesn't look likely in the near future.

--
David Goodger <http://python.net/~goodger>

Mark Nodine wrote:

> Felix Wiemann wrote:
>
>> BerliOS seems to be ignoring requests for WebDAV support.
>
> Is the problem that they are not running an Apache server?

Maybe.  They're running Apache 1.3.27, and Subversion requires Apache 2
for WebDAV access.

> Would the existence proof of a would-be contributor who is unable to
> do so without WebDAV access help to turn up the heat?

Possibly, but others have asked before.  You could add your comment to
<https://developer.berlios.de/feature/?func=detailfeature&feature_id=471&group_id=1>
and write email to the admins.

The BerliOS admins seem to be either stupid or overloaded.  They didn't
respond to
<https://developer.berlios.de/bugs/?func=detailbug&group_id=1&bug_id=3524>
either, let alone fix it; that's why we currently don't have real names
in our commit emails.  :-/

-- 
For private mail please ensure that the header contains 'Felix Wiemann'.

http://www.ososo.de/