[pacman-dev] [PATCH 0/7] integrate test suite with automake

[Allan McRae]

On 05/08/13 14:16, Andrew Gregory wrote:
> On 08/05/13 at 10:52am, Allan McRae wrote:
>> On 02/08/13 22:34, Andrew Gregory wrote:
>>> This patchset converts the output of all of our tests to tap [1] and fully
>>> integrates them with automake so that tests can be run in parallel with `make
>>> check`.  The test suite may also be run with other test harnesses such as
>>> perl's prove which can do such interesting things as remember which tests
>>> failed and run only those on subsequent invocations.  The documentation for
>>> integrating tests is here [2].
>>>
>>> [1] http://podwiki.hexten.net/TAP/TAP.html?page=TAP
>>> [2] http://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
>>>
>>
>> Have you any ideas on how to fix the "unexpected" pass on the time test
>> for x86_64 to not have the test suite return non-zero?  I believe this
>> is essential.
>>
>> Allan
> 
> I think that "unexpected" passes are rightly considered failures.  The test
> should reflect what we actually expect to happen.  We should either update the
> test so that it succeeds or fails uniformly on all systems or set expectfailure
> only on systems where we actually expect it to fail.  Personally, I would
> prefer that the test use the maximum values that the testing system could be
> expected to support and unset expectfailure, but the easier solution is to just
> set expectfailure only on 32 bit systems.

Setting expected failure on 32bit systems would actually be my preferred
solution in this case.  Can our test suite handle that?