[arch-releng] release cycle & testing procedures

[Dieter Plaetinck]

Beloved colleagues,

I made a section for releng @ 
http://wiki.archlinux.org/index.php/DeveloperWiki#Release_Engineering
but most of these things need to be discussed before they are documented :)
So here we go:

release cycle?:
2-3 months. based on kernel releases.
* upstream releases a kernel
* arch dev(s) bring it into testing
* arch-releng builds an iso (using all core packages with just kernel 
from testing? or all testing packages? Note that when the kernel goes 
into core later on, not all packages will be moved at the same point.  
so we should test what we intend to release, and when we release, we 
should only use core packages.).  we could also wait until all we need 
(mostly the kernel) gets into core, but I think this wastes too much time.
* we test it.
* if the kernel goes into core, our are tests are satisfactory and there 
are no known showstoppers/bugs (other then tiny ones). we can continue:
* prepare torrent, upload to our mirror (+give mirrors time to sync, how 
long?), prepare announcement.
* official release
* if major issues are discovered after official release, we fix them and 
bump the the revision (eg 2009.01-2)
* time for beer, games and misc bugfixing/featurerequests.
* restart cycle

questions:
1) do we do testing releases? right now lots of people are eager for a 
release, even lesser tested ones.  But this is probably 99.9% caused by 
the lust for ext4.   if we plan to release every 2-3 months, there 
probably is no need for testing releases.
on the other hand, the community could be useful for helping out testing 
our stuff/giving feedback, and it isn't much effort to do a testing 
release i think.

2) how have iso's been tested previously?  It seems reasonable to assume 
that if we only use core packages, most bugs should be squased already.  
however, installing a new system causes different operations then 
booting/using an existing one.
So I suggest we do a few installations using some different features 
(ftp install vs cd, autopartitioning vs manual, create a lot of 
different filesystems and dm-mapper thingies like lvm and dm_crypt, etc) 
and on both supported arches.


3) Automated testing: a feature in aif's pipeline is automatic 
(scripted) installations. using this, we should be able to save 
ourselves some time.
I'm thinking about something like this:
* boot a VM
* start aif -p automatic with the appropriate config. test lots of 
different things (see question 2). we can even go further and basically 
automatically test all features that AIF offers ( except the UI)
* after installation, aif puts a script on the target system, and makes 
it execute on boot
* reboot.
* system boots and executes the script. this scripts does some network 
checks, check if pacman -Sy works, check lvdisplay, df -h, /etc/mtab, 
etc. mail the results to releng.