On Fri, Jan 23, 2009 at 12:25 PM, Dieter Plaetinck
Beloved colleagues,
I made a section for releng @ http://wiki.archlinux.org/index.php/DeveloperWiki#Release_Engineering but most of these things need to be discussed before they are documented :) So here we go:
release cycle?: 2-3 months. based on kernel releases. * upstream releases a kernel * arch dev(s) bring it into testing * arch-releng builds an iso (using all core packages with just kernel from testing? or all testing packages? Note that when the kernel goes into core later on, not all packages will be moved at the same point. so we should test what we intend to release, and when we release, we should only use core packages.). we could also wait until all we need (mostly the kernel) gets into core, but I think this wastes too much time. * we test it. * if the kernel goes into core, our are tests are satisfactory and there are no known showstoppers/bugs (other then tiny ones). we can continue: * prepare torrent, upload to our mirror (+give mirrors time to sync, how long?), prepare announcement. * official release * if major issues are discovered after official release, we fix them and bump the the revision (eg 2009.01-2) * time for beer, games and misc bugfixing/featurerequests. * restart cycle
Regarding packages: I'd say the initial ISO is built with full testing enabled. Then later after we get things tested (a move from "alpha" to "beta" status or something), we plan a core freeze for when all necessary packages will be in core, so that we can build the new ISOs. Other than that, this sounds good.
1) do we do testing releases? right now lots of people are eager for a release, even lesser tested ones. But this is probably 99.9% caused by the lust for ext4. if we plan to release every 2-3 months, there probably is no need for testing releases. on the other hand, the community could be useful for helping out testing our stuff/giving feedback, and it isn't much effort to do a testing release i think.
I think the initial alpha/beta to a smaller audience should cover us here - the devs and this list should cover most basic testing.
2) how have iso's been tested previously? It seems reasonable to assume that if we only use core packages, most bugs should be squased already. however, installing a new system causes different operations then booting/using an existing one. So I suggest we do a few installations using some different features (ftp install vs cd, autopartitioning vs manual, create a lot of different filesystems and dm-mapper thingies like lvm and dm_crypt, etc) and on both supported arches.
To be absolutely certain of quality, we should really create a checklist of things to test (say, a wiki page). We can update it as we get new features. In the past, people would just boot it and do their thing, with regard to testing the installer.
3) Automated testing: a feature in aif's pipeline is automatic (scripted) installations. using this, we should be able to save ourselves some time. I'm thinking about something like this: * boot a VM * start aif -p automatic with the appropriate config. test lots of different things (see question 2). we can even go further and basically automatically test all features that AIF offers ( except the UI) * after installation, aif puts a script on the target system, and makes it execute on boot * reboot. * system boots and executes the script. this scripts does some network checks, check if pacman -Sy works, check lvdisplay, df -h, /etc/mtab, etc. mail the results to releng.
This would be awesome if we could automate this! I LOVE automated testing.