On Fri, Jan 23, 2009 at 12:25 PM, Dieter Plaetinck <dieter@plaetinck.be> wrote:
  

Beloved colleagues,

I made a section for releng @
http://wiki.archlinux.org/index.php/DeveloperWiki#Release_Engineering
but most of these things need to be discussed before they are documented :)
So here we go:

release cycle?:
2-3 months. based on kernel releases.
* upstream releases a kernel
* arch dev(s) bring it into testing
* arch-releng builds an iso (using all core packages with just kernel from
testing? or all testing packages? Note that when the kernel goes into core
later on, not all packages will be moved at the same point.  so we should
test what we intend to release, and when we release, we should only use core
packages.).  we could also wait until all we need (mostly the kernel) gets
into core, but I think this wastes too much time.
* we test it.
* if the kernel goes into core, our are tests are satisfactory and there are
no known showstoppers/bugs (other then tiny ones). we can continue:
* prepare torrent, upload to our mirror (+give mirrors time to sync, how
long?), prepare announcement.
* official release
* if major issues are discovered after official release, we fix them and
bump the the revision (eg 2009.01-2)
* time for beer, games and misc bugfixing/featurerequests.
* restart cycle
    

Regarding packages: I'd say the initial ISO is built with full testing
enabled. Then later after we get things tested (a move from "alpha" to
"beta" status or something), we plan a core freeze for when all
necessary packages will be in core, so that we can build the new ISOs.

Other than that, this sounds good.
  

1) do we do testing releases? right now lots of people are eager for a
release, even lesser tested ones.  But this is probably 99.9% caused by the
lust for ext4.   if we plan to release every 2-3 months, there probably is
no need for testing releases.
on the other hand, the community could be useful for helping out testing our
stuff/giving feedback, and it isn't much effort to do a testing release i
think.
    

I think the initial alpha/beta to a smaller audience should cover us
here - the devs and this list should cover most basic testing.

  

2) how have iso's been tested previously?  It seems reasonable to assume
that if we only use core packages, most bugs should be squased already.
 however, installing a new system causes different operations then
booting/using an existing one.
So I suggest we do a few installations using some different features (ftp
install vs cd, autopartitioning vs manual, create a lot of different
filesystems and dm-mapper thingies like lvm and dm_crypt, etc) and on both
supported arches.
    

To be absolutely certain of quality, we should really create a
checklist of things to test (say, a wiki page). We can update it as we
get new features. In the past, people would just boot it and do their
thing, with regard to testing the installer.

  

3) Automated testing: a feature in aif's pipeline is automatic (scripted)
installations. using this, we should be able to save ourselves some time.
I'm thinking about something like this:
* boot a VM
* start aif -p automatic with the appropriate config. test lots of different
things (see question 2). we can even go further and basically automatically
test all features that AIF offers ( except the UI)
* after installation, aif puts a script on the target system, and makes it
execute on boot
* reboot.
* system boots and executes the script. this scripts does some network
checks, check if pacman -Sy works, check lvdisplay, df -h, /etc/mtab, etc.
mail the results to releng.
    

This would be awesome if we could automate this! I LOVE automated testing.