[arch-releng] release cycle & testing procedures

[Aaron Griffin]

On Fri, Jan 23, 2009 at 12:25 PM, Dieter Plaetinck <dieter at plaetinck.be> wrote:
> Beloved colleagues,
>
> I made a section for releng @
> http://wiki.archlinux.org/index.php/DeveloperWiki#Release_Engineering
> but most of these things need to be discussed before they are documented :)
> So here we go:
>
> release cycle?:
> 2-3 months. based on kernel releases.
> * upstream releases a kernel
> * arch dev(s) bring it into testing
> * arch-releng builds an iso (using all core packages with just kernel from
> testing? or all testing packages? Note that when the kernel goes into core
> later on, not all packages will be moved at the same point.  so we should
> test what we intend to release, and when we release, we should only use core
> packages.).  we could also wait until all we need (mostly the kernel) gets
> into core, but I think this wastes too much time.
> * we test it.
> * if the kernel goes into core, our are tests are satisfactory and there are
> no known showstoppers/bugs (other then tiny ones). we can continue:
> * prepare torrent, upload to our mirror (+give mirrors time to sync, how
> long?), prepare announcement.
> * official release
> * if major issues are discovered after official release, we fix them and
> bump the the revision (eg 2009.01-2)
> * time for beer, games and misc bugfixing/featurerequests.
> * restart cycle

Regarding packages: I'd say the initial ISO is built with full testing
enabled. Then later after we get things tested (a move from "alpha" to
"beta" status or something), we plan a core freeze for when all
necessary packages will be in core, so that we can build the new ISOs.

Other than that, this sounds good.

> 1) do we do testing releases? right now lots of people are eager for a
> release, even lesser tested ones.  But this is probably 99.9% caused by the
> lust for ext4.   if we plan to release every 2-3 months, there probably is
> no need for testing releases.
> on the other hand, the community could be useful for helping out testing our
> stuff/giving feedback, and it isn't much effort to do a testing release i
> think.

I think the initial alpha/beta to a smaller audience should cover us
here - the devs and this list should cover most basic testing.

> 2) how have iso's been tested previously?  It seems reasonable to assume
> that if we only use core packages, most bugs should be squased already.
>  however, installing a new system causes different operations then
> booting/using an existing one.
> So I suggest we do a few installations using some different features (ftp
> install vs cd, autopartitioning vs manual, create a lot of different
> filesystems and dm-mapper thingies like lvm and dm_crypt, etc) and on both
> supported arches.

To be absolutely certain of quality, we should really create a
checklist of things to test (say, a wiki page). We can update it as we
get new features. In the past, people would just boot it and do their
thing, with regard to testing the installer.

> 3) Automated testing: a feature in aif's pipeline is automatic (scripted)
> installations. using this, we should be able to save ourselves some time.
> I'm thinking about something like this:
> * boot a VM
> * start aif -p automatic with the appropriate config. test lots of different
> things (see question 2). we can even go further and basically automatically
> test all features that AIF offers ( except the UI)
> * after installation, aif puts a script on the target system, and makes it
> execute on boot
> * reboot.
> * system boots and executes the script. this scripts does some network
> checks, check if pacman -Sy works, check lvdisplay, df -h, /etc/mtab, etc.
> mail the results to releng.

This would be awesome if we could automate this! I LOVE automated testing.