[arch-releng] release cycle & testing procedures

Dieter Plaetinck dieter at plaetinck.be
Fri Jan 23 17:04:00 EST 2009


Aaron Griffin wrote:
> On Fri, Jan 23, 2009 at 12:25 PM, Dieter Plaetinck <dieter at plaetinck.be> wrote:
>   
>> Beloved colleagues,
>>
>> I made a section for releng @
>> http://wiki.archlinux.org/index.php/DeveloperWiki#Release_Engineering
>> but most of these things need to be discussed before they are documented :)
>> So here we go:
>>
>> release cycle?:
>> 2-3 months. based on kernel releases.
>> * upstream releases a kernel
>> * arch dev(s) bring it into testing
>> * arch-releng builds an iso (using all core packages with just kernel from
>> testing? or all testing packages? Note that when the kernel goes into core
>> later on, not all packages will be moved at the same point.  so we should
>> test what we intend to release, and when we release, we should only use core
>> packages.).  we could also wait until all we need (mostly the kernel) gets
>> into core, but I think this wastes too much time.
>> * we test it.
>> * if the kernel goes into core, our are tests are satisfactory and there are
>> no known showstoppers/bugs (other then tiny ones). we can continue:
>> * prepare torrent, upload to our mirror (+give mirrors time to sync, how
>> long?), prepare announcement.
>> * official release
>> * if major issues are discovered after official release, we fix them and
>> bump the the revision (eg 2009.01-2)
>> * time for beer, games and misc bugfixing/featurerequests.
>> * restart cycle
>>     
>
> Regarding packages: I'd say the initial ISO is built with full testing
> enabled. Then later after we get things tested (a move from "alpha" to
> "beta" status or something), we plan a core freeze for when all
> necessary packages will be in core, so that we can build the new ISOs.
>
> Other than that, this sounds good.
>   
do I understand correctly that you want to move all packages from 
testing used by the alpha iso to core before releasing the "beta"?
eg alpha and beta offer exactly the same versions of all packages?  Or 
is there some room for not moving some packages?
>   
>> 1) do we do testing releases? right now lots of people are eager for a
>> release, even lesser tested ones.  But this is probably 99.9% caused by the
>> lust for ext4.   if we plan to release every 2-3 months, there probably is
>> no need for testing releases.
>> on the other hand, the community could be useful for helping out testing our
>> stuff/giving feedback, and it isn't much effort to do a testing release i
>> think.
>>     
>
> I think the initial alpha/beta to a smaller audience should cover us
> here - the devs and this list should cover most basic testing.
>
>   
>> 2) how have iso's been tested previously?  It seems reasonable to assume
>> that if we only use core packages, most bugs should be squased already.
>>  however, installing a new system causes different operations then
>> booting/using an existing one.
>> So I suggest we do a few installations using some different features (ftp
>> install vs cd, autopartitioning vs manual, create a lot of different
>> filesystems and dm-mapper thingies like lvm and dm_crypt, etc) and on both
>> supported arches.
>>     
>
> To be absolutely certain of quality, we should really create a
> checklist of things to test (say, a wiki page). We can update it as we
> get new features. In the past, people would just boot it and do their
> thing, with regard to testing the installer.
>
>   
Yep, I'll do that
>> 3) Automated testing: a feature in aif's pipeline is automatic (scripted)
>> installations. using this, we should be able to save ourselves some time.
>> I'm thinking about something like this:
>> * boot a VM
>> * start aif -p automatic with the appropriate config. test lots of different
>> things (see question 2). we can even go further and basically automatically
>> test all features that AIF offers ( except the UI)
>> * after installation, aif puts a script on the target system, and makes it
>> execute on boot
>> * reboot.
>> * system boots and executes the script. this scripts does some network
>> checks, check if pacman -Sy works, check lvdisplay, df -h, /etc/mtab, etc.
>> mail the results to releng.
>>     
>
> This would be awesome if we could automate this! I LOVE automated testing.
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.archlinux.org/pipermail/arch-releng/attachments/20090123/41077ac8/attachment-0001.htm>


More information about the arch-releng mailing list