On Sat, 19 Feb 2011 23:58:23 +0100 Tom Willemsen <tom.willemsen@archlinux.us> wrote:
well I think it would be best to have a table that shows for each value of each criterion, the last succesful state (if known)
so, for example:
criterion ---- last known state Arch: - i686 OK on 2011.02.10 (tested by $user) - x86_64 OK on 2011.02.05 (tested by $user) - dual/i686 not tested yet - dual/x86_64 FAILED on 2011.01.25 (tested by $user)
Image type: - Core OK on 2011.02.10 (tested by $user) - Netinstall OK on 2011.01.18 (tested by $user)
and so on (see my first mail for all criteria)
So, since every test report is a combination of reports for specific things (like "i tested an install with the i686 core image, with manual partitioning, ntp clock setting, etc etc and it worked fine"), but it will cause updates for various properties in the table above.
Dieter
Won't that get troublesome though? I mean, someone tests it with i686, core, automatic partitioning and it works, but then someone tests it with i686, net, automatic partioning and it doesn't. At this point i686 will have worked, core will have worked and automatic partitioning will have worked, but only net won't have. But then someone might test it with x86_64 and that does work (ok so maybe using the core/net difference might be a bad idea? sorry) then net will have worked too, but not on i686 which might not be apparent from this data. Or am I just being picky?
Tom
Well, the parameters are usually independent from each other. for example the netinstall code is the same, whether you use the i686 image or the x86_64 one, and whether you use an i686 repo or an x86_64 one. there might be some correlations between parameters but if we can already make the webapp work and do the report as described (without showing "other parameters used in the test case that failed"), that would already be a great step. And the idea is to reach 0 failures anyway, so as soon as something fails, we need to debug why and fix it, so that all entries become 'OK' again for later images. The user can also supply comments (and hopefully he will make a ticket on the bugtracker with detailed info). If you want, you can provide further interfaces to show complete reports, link to those from the main table and so on, but let's first get the basics working. Dieter