openvpn-announce Mailing List for OpenVPN (Page 3)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi,

> So just trying to hijack this discussion which is to be found a few more
> places elsewhere in this mail thread.  No need to let this discussion
> run longer.
> 
> There are several area where we definitely can improve the release
> process.  Last round where we managed to mess up the 2.3.15 release, so
> I wrote a brand new "prepare release tarballs" script, which also
> handles the signing.  This script _was_ used to produce the files to be
> pushed out for the 2.4.3/2.3.17 releases.
> 
> But for reasons unknown to me, those tarballs got re-created somewhere
> later in the release chain.  The contents of all tarballs are

The tarballs were created by my release script which is available on our
internal Bitbucket. I always try to be very careful in what I upload,
but some mistakes seem unavoidable due to the really complex nature of
the release process.

My release script predates the script you wrote for OpenVPN 2.4.2
release. My script operates on the assumption that a fully up-to-date
Git repository with release tags and everything is available. This has
generally been the case, as we've usually updated Git first and _then_
started the release machinery.

For me it would be easiest if I could handle the entire release process
starting from an fully up-to-date Git repository. That would allow my
script(s) to automatically handle most technical parts of the release:

- Produce the tarballs
- Sign them with the sec...@op... key
- Verify the signatures
- Generate changelogs for Trac and Debian packages
- Generate a man-page suitable for copy-and-pasting in Trac
- Generate a CloudFlare flush file
  - Copy-and-paste to CloudFlare to flush caches for all the files;
    this is a routine release procedure nowadays
- Push everything to the download servers
- Download everything from the download servers _and_ verify the signatures

All of this is already handled and works well enough. I _can_ use
tarballs as the source, but I also need a Git repo to get Git logs, so
the tarballs are a bit redundant.

> essentially the same, but due to the "nice" artefact that the tar format
> is non-deterministic on the output, even though the input is the same,
> that begins to prepare the stage for this chaos.  Especially when what
> is being uploaded is partly from the initial run and then some files
> from a different run.
> 
> All that is history now.  Now we need to look forward.  Many good points
> have been raised.
> 
> - Do we need .tar.gz and .zip files?  Where and why?
>   The fewer source tarballs we need to handle, the less chance for
>   errors

I would love if we could drop everything except tar.xz. This way the
amount of files and signatures on the download page would drop
significantly. I think it would be reasonable to expect that people who
need OpenVPN _sources_ on Windows are able to extract tar.xz, especially
if we document how it can be done.

> 
> - Improve Makefile.am to not generate dist-gz files when running
>   distcheck.  The distcheck run often provides very good indicator if we
>   have packaged all the needed files in the source tarball.  If this
>   doesn't pass, something is really wrong.
> 
> - Do we really need to re-create the source tarballs which the new
>   ./dev-tools/gen-release-tarballs.sh?  Why?

No. But another question is whether we need gen-release-tarballs.sh
which implements a limited subset of the release script I had written
earlier? The gen-release-tarballs.sh came as a surprise to me -
otherwise I would told you that what it does is already covered.

Before answering the question, though, we should figure out our overall
release strategy.

> 
> - What can be done with Cloudflare to fully ensure their caches are
>   truly purged when we ask for it?  As Jonathan noticed, their caches
>   are tightly connected to the web browser and have a non-deterministic
>   behaviour across browsers, even on the same computer.
> 

As mentioned above, I've routinely purge CloudFlare caches for all
release files on every release for a long time. Initially this was
because CloudFlare cached 404's for some people. Occasionally issues
arose when a wrong version of a file ended up on the download server.

Regardless of all these precautions we still get some CloudFlare-related
complaints on every release.

> - What else in the release process can be automated and put into a
>   script?  This to ensure consistency between all releases we do.

The following things rob a surprising amount of time from me during a
release:

1. Producing release announcement in three different formats:
   - Download page
   - Forums
   - Mailing list
2. Playing with various Git repos
   - easy-rsa-old, openvpn-build, openvpn-gui, openvpnserv2
   - For each of them in one or more branches
     - Pushing to my fork
     - Pushing version changes and tags to upstream repo
       - Must ensure that what is pushed is the bare minimum
3. Building and testing Debian packages
4) Building and testing Windows installers
5) Playing with CloudFlare caches

Producing release announcements (1) from a template would help quite a
bit actually, as the announcements are generally very similar to each
other. This script could potentially be public.

Parts of 2) have been automated by my release script, but there is still
room for improvement. This also could potentially be public.

Debian package (3) building generally goes smoothly, but there are
occasional hickups when something has changed somewhere (e.g. OpenVPN or
the underlying OS). Plus building tons of packages simply takes a lot of
time. This is already public, but could be automated further.

Windows installer building (4) is fairly straightforward and testing has
been automated using openvpn-windows-test (see GitHub). Still several
different versions have to be built atm:

- 2.4.x combined (32/64-bit) installer for Vista+
- 2.3.x installers (32/64-bit) for Vista+
- 2.3.x installers (32/64-bit) for XP

Building and testing these takes a lot of time and care. All the scripts
are public already, but further automation is possible.

CloudFlare (5) cache clearing could probably be automated fairly easily.
A separate Python script, for example, could be used. If this part is
made generic enough it could be made public.

> - We need to write down a proper check-list of all the steps needed
>   for a release, including putting a clear responsibility for each
>   release.  This list must also mention which scripts to be run.  Again,
>   automation is key to reduce the risk for errors.

We have a pretty thorough internal checklist in JIRA.

> - Consider how many who really needs to be involved in producing a
>   release.  More chefs in a kitchen can result in great food, but it can
>   also end up quite messy.

Agreed.

Many of the tasks in the JIRA ticket do not require any special access
to OpenVPN Technologies internal services (CloudFlare) or servers
(download/build/management servers). So those tasks could potentially be
handled by community developers. But would that make sense? Would it not
be easier to handle the entire release process from one point and just
ensure that the process can be replicated by more than one person (an
employee)?

What we could do is split the release into logical single-purpose steps
each of which is handled by a separate script. The scripts which would
not need access to OpenVPN Tech servers or CloudFlare could then be
published on GitHub. Or, if the scripts are generic enough they could
(potentially) be used by others by simply modifying a configuration file
or command-line options.

I've done this for some of the larger scripts I use for releases:

- openvpn-windows-test (the Powershell test suite)
- sbuild_wrapper (used to produce Debian packages)

> - At the same time, ensure we don't end up in a "single point of
>   failure".  More of us core developers need to be able to step in for
>   others, and still be able to produce a release without errors.  This
>   can be the end result if we have proper scripts, both for automated
>   and manual tasks.

I think you are the best fit for the role :). You have a good
understanding of the release process and have or can be granted the
access that is needed to handle all the release steps.

> My intention with these points are primarily "food for thought".  I
> don't fully believe it will be easy to have a well structured debate
> about the complete release process in a mailing list thread.
> 
> So I suggest we take a few weeks holiday, let this sink in, and then we
> can schedule a meeting some time in August where we discuss these
> issues.  And lets hope we don't need to rush yet another release before
> August :)
> 
> 

Makes sense.

-- 
Samuli Seppänen
Community Manager
OpenVPN Technologies, Inc

irc freenode net: mattock

2002	Jan	Feb	Mar	Apr (2)	May (1)	Jun (1)	Jul (3)	Aug	Sep (1)	Oct	Nov	Dec
2003	Jan	Feb	Mar	Apr	May (3)	Jun	Jul (2)	Aug (1)	Sep	Oct (1)	Nov (2)	Dec (1)
2004	Jan	Feb	Mar	Apr	May (1)	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2005	Jan (1)	Feb	Mar	Apr (1)	May	Jun	Jul	Aug (2)	Sep	Oct	Nov (1)	Dec
2006	Jan	Feb	Mar	Apr (1)	May	Jun	Jul	Aug	Sep (2)	Oct (1)	Nov	Dec
2008	Jan (1)	Feb	Mar	Apr	May (1)	Jun	Jul	Aug (1)	Sep (1)	Oct	Nov	Dec
2009	Jan	Feb	Mar	Apr	May (1)	Jun	Jul	Aug	Sep	Oct	Nov (1)	Dec (1)
2010	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug (1)	Sep	Oct	Nov (1)	Dec (1)
2011	Jan	Feb	Mar (2)	Apr (1)	May	Jun	Jul (1)	Aug	Sep	Oct	Nov	Dec (1)
2012	Jan	Feb (1)	Mar	Apr	May	Jun	Jul (2)	Aug	Sep (1)	Oct (1)	Nov	Dec (1)
2013	Jan (2)	Feb (1)	Mar (1)	Apr	May	Jun (1)	Jul	Aug	Sep	Oct	Nov	Dec
2014	Jan	Feb	Mar	Apr (1)	May (1)	Jun (1)	Jul	Aug (3)	Sep	Oct (2)	Nov (1)	Dec (1)
2015	Jan	Feb	Mar (1)	Apr	May	Jun (1)	Jul (1)	Aug (1)	Sep	Oct	Nov	Dec (1)
2016	Jan (1)	Feb (1)	Mar	Apr	May (2)	Jun	Jul	Aug (1)	Sep	Oct (1)	Nov (3)	Dec (4)
2017	Jan	Feb	Mar (2)	Apr	May (4)	Jun (3)	Jul (1)	Aug	Sep (4)	Oct	Nov	Dec
2018	Jan	Feb	Mar (1)	Apr (1)	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2019	Jan	Feb (1)	Mar	Apr (1)	May	Jun	Jul	Aug	Sep	Oct (2)	Nov	Dec
2020	Jan	Feb	Mar	Apr (1)	May	Jun	Jul	Aug (1)	Sep (4)	Oct (2)	Nov	Dec (1)
2021	Jan	Feb (1)	Mar	Apr (2)	May	Jun (1)	Jul	Aug	Sep	Oct (1)	Nov	Dec (1)
2022	Jan	Feb	Mar (2)	Apr	May (1)	Jun	Jul	Aug	Sep	Oct	Nov (2)	Dec (3)
2023	Jan (2)	Feb (1)	Mar (2)	Apr (1)	May (1)	Jun (1)	Jul	Aug (1)	Sep	Oct	Nov (3)	Dec
2024	Jan	Feb (1)	Mar (1)	Apr	May	Jun (1)	Jul (1)	Aug	Sep	Oct	Nov	Dec

openvpn-announce Mailing List for OpenVPN (Page 3)

Robust and flexible VPN network tunnelling

openvpn-announce — OpenVPN announcements