Re: [pacman-dev] [PATCH v2] Add testrunner for makepkg-template

On 12/21/14 at 04:43pm, Florian Pritz wrote:

Signed-off-by: Florian Pritz ---

v2: - use a temp dir for output files - mark the exit code in the expected_output file with "exitcode: " (tests are adjusted on my working branch)

Makefile.am | 2 + test/scripts/Makefile.am | 1 + test/scripts/makepkg-template_test.sh | 89 +++++++++++++++++++++++++++++++++++ 3 files changed, 92 insertions(+) create mode 100755 test/scripts/makepkg-template_test.sh

diff --git a/Makefile.am b/Makefile.am index e9b3dfa..ab669ef 100644 --- a/Makefile.am +++ b/Makefile.am @@ -28,6 +28,7 @@ $(top_srcdir)/test/pacman/tests/TESTS: $(wildcard test/pacman/tests/*.py*)

TESTS = test/scripts/parseopts_test.sh \ test/scripts/human_to_size_test.sh \ + test/scripts/makepkg-template_test.sh \ test/scripts/pacman-db-upgrade-v9.py \ test/util/pacsorttest.sh \ test/util/vercmptest.sh @@ -37,6 +38,7 @@ TEST_SUITE_LOG = test/test-suite.log TEST_EXTENSIONS = .py AM_TESTS_ENVIRONMENT = \ PMTEST_UTIL_DIR=$(top_builddir)/src/util/; export PMTEST_UTIL_DIR; \ + PMTEST_SCRIPT_DIR=$(top_srcdir)/scripts/; export PMTEST_SCRIPT_DIR; \ PMTEST_SCRIPTLIB_DIR=$(top_srcdir)/scripts/library/; export PMTEST_SCRIPTLIB_DIR; LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \ $(top_srcdir)/build-aux/tap-driver.sh diff --git a/test/scripts/Makefile.am b/test/scripts/Makefile.am index 8d6bc84..ed1a951 100644 --- a/test/scripts/Makefile.am +++ b/test/scripts/Makefile.am @@ -1,6 +1,7 @@ check_SCRIPTS = \ parseopts_test.sh \ pacman-db-upgrade-v9.py \ + makepkg-template_test.sh \ human_to_size_test.sh

noinst_SCRIPTS = $(check_SCRIPTS) diff --git a/test/scripts/makepkg-template_test.sh b/test/scripts/makepkg-template_test.sh new file mode 100755 index 0000000..5242f2c --- /dev/null +++ b/test/scripts/makepkg-template_test.sh @@ -0,0 +1,89 @@ +#!/bin/bash + +declare -i testcount=0 fail=0 pass=0 total=0 + +# source the library function +scriptdir=${1:-${PMTEST_SCRIPT_DIR}} +if [[ -z $scriptdir || ! -f $scriptdir/makepkg-template ]]; then + printf "Bail out! makepkg-template executable (%s) could not be located\n" "${scriptdir}/makepkg-template" + exit 1 +fi

This does not work the same as our other tests which allow passing the executable as the first argument.

+ +testdir="${0%/*}/makepkg-template-tests" + +total=$(find "$testdir" -maxdepth 1 -mindepth 1 -type d | wc -l) + +run_test() { + local testcase=$1 exitcode + local -i failed=0 + + (( ++testcount )) + + local -a arguments + + if [[ -f "$testdir/$testcase/arguments" ]]; then + arguments=($(cat "$testdir/$testcase/arguments")) + fi + + mkdir "$TMPDIR/$testcase" + + "$scriptdir/makepkg-template" \

makepkg-template uses translated messages; if you're going to check its output you need to set the locale.

+ --template-dir "$testdir/$testcase/templates" \ + -p "$testdir/$testcase/PKGBUILD" \ + -o "$TMPDIR/$testcase/result" \ + &> "$TMPDIR/$testcase/output_full" "${arguments[@]}" + exitcode=$? + + sed "$TMPDIR/$testcase/output_full" >"$TMPDIR/$testcase/output" \ + -e 's|./test/scripts/|./|' \

That should be $testdir or some variation of it.

+ -e 's| at '$scriptdir'/makepkg-template line [0-9]\+\.$||'

makepkg-template should be updated to not include line numbers in error messages rather than trying to filter them out.

+ printf "exitcode: %s\n" "$exitcode" >> "$TMPDIR/$testcase/output"

It would be better if we could test the exit code, output, and result independently.

+ + if [[ -f "$TMPDIR/$testcase/result" ]]; then + if ! diff -u "$TMPDIR/$testcase/result" "$testdir/$testcase/expected_result"; then

The diff output needs to be marked as diagnostic output. I'm working on a TAP generator for bash that we can use that will do that for us as well as taking care of all the counting.

+ failed=1 + fi + else + if [[ -f "$testdir/$testcase/expected_result" ]]; then + echo "$testcase: Found expected_result file but no result file" + failed=1 + fi + fi + + if ! diff -u "$TMPDIR/$testcase/output" "$testdir/$testcase/expected_output"; then + failed=1 + fi + + if ((!failed)); then + (( ++pass )) + printf "ok %d - %s\n" "$testcount" "$input"

I think you mean s/input/testcase/

+ else + (( ++fail )) + printf "not ok %d - %s\n" "$testcount" "$input" + printf '# [TEST %3s]: FAIL\n' "$testcount"

Redundant failure message.

+ fi +} + +summarize() { + rm -rf "$TMPDIR" + + if (( !fail )); then + printf '# All %s tests successful\n\n' "$testcount" + exit 0 + else + printf '# %s of %s tests failed\n\n' "$fail" "$testcount" + exit 1 + fi +} +TMPDIR="$(mktemp -d "/tmp/${0##*/}.XXXXXX")" +trap 'summarize' EXIT + +printf '# Beginning makepkg-template tests\n'

I know we have these messages in other tests, but they are completely useless, let's not add any more.

+ +echo "1..$total" + +for dir in "$testdir/"*; do + if [[ -d "$dir" ]]; then + run_test "${dir##*/}" + fi +done -- 2.2.1