[pacman-dev] [PATCH 0/7] integrate test suite with automake

[Andrew Gregory]

On 08/12/13 at 09:10pm, Allan McRae wrote:
> On 05/08/13 14:47, Andrew Gregory wrote:
> > On 08/05/13 at 02:18pm, Allan McRae wrote:
> >> On 05/08/13 14:16, Andrew Gregory wrote:
> >>> On 08/05/13 at 10:52am, Allan McRae wrote:
> >>>> On 02/08/13 22:34, Andrew Gregory wrote:
> >>>>> This patchset converts the output of all of our tests to tap [1] and fully
> >>>>> integrates them with automake so that tests can be run in parallel with `make
> >>>>> check`.  The test suite may also be run with other test harnesses such as
> >>>>> perl's prove which can do such interesting things as remember which tests
> >>>>> failed and run only those on subsequent invocations.  The documentation for
> >>>>> integrating tests is here [2].
> >>>>>
> >>>>> [1] http://podwiki.hexten.net/TAP/TAP.html?page=TAP
> >>>>> [2] http://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
> >>>>>
> >>>>
> >>>> Have you any ideas on how to fix the "unexpected" pass on the time test
> >>>> for x86_64 to not have the test suite return non-zero?  I believe this
> >>>> is essential.
> >>>>
> >>>> Allan
> >>>
> >>> I think that "unexpected" passes are rightly considered failures.  The test
> >>> should reflect what we actually expect to happen.  We should either update the
> >>> test so that it succeeds or fails uniformly on all systems or set expectfailure
> >>> only on systems where we actually expect it to fail.  Personally, I would
> >>> prefer that the test use the maximum values that the testing system could be
> >>> expected to support and unset expectfailure, but the easier solution is to just
> >>> set expectfailure only on 32 bit systems.
> >>
> >> Setting expected failure on 32bit systems would actually be my preferred
> >> solution in this case.  Can our test suite handle that?
> > 
> > I don't have any 32-bit systems readily available to test it at the moment, but
> > checking either platform.architecture [1] or sys.maxsize [2] should be
> > sufficient.
> > 
> > [1] http://docs.python.org/2/library/platform.html#platform.architecture
> > [2] http://docs.python.org/2/library/sys.html#sys.maxsize
> > 
> 
> I guess I can test this in a chroot (or you could...).
> 
> It also looks like .gitignore needs updated:
> 
> #       test-suite.log
> #       test/pacman/tests/clean001.log
> #       test/pacman/tests/clean001.trs
> #       test/pacman/tests/clean002.log
> #       test/pacman/tests/clean002.trs
> #       test/pacman/tests/clean003.log
> #       test/pacman/tests/clean003.trs
> #       test/pacman/tests/clean004.log
> #       test/pacman/tests/clean004.trs
> #       test/pacman/tests/clean005.log

Erm, I did update .gitignore...  Did you by any chance run make check
with these patches then switch to a different branch?  Otherwise
I have no idea why those would show up.

apg