[pacman-dev] [PATCH v2] Add testrunner for makepkg-template

[Andrew Gregory]

On 12/21/14 at 04:43pm, Florian Pritz wrote:
> Signed-off-by: Florian Pritz <bluewind at xinu.at>
> ---
> 
> v2:
>  - use a temp dir for output files
>  - mark the exit code in the expected_output file with "exitcode: "
>    (tests are adjusted on my working branch)
> 
>  Makefile.am                           |  2 +
>  test/scripts/Makefile.am              |  1 +
>  test/scripts/makepkg-template_test.sh | 89 +++++++++++++++++++++++++++++++++++
>  3 files changed, 92 insertions(+)
>  create mode 100755 test/scripts/makepkg-template_test.sh
> 
> diff --git a/Makefile.am b/Makefile.am
> index e9b3dfa..ab669ef 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -28,6 +28,7 @@ $(top_srcdir)/test/pacman/tests/TESTS: $(wildcard test/pacman/tests/*.py*)
>  
>  TESTS =  test/scripts/parseopts_test.sh \
>  				 test/scripts/human_to_size_test.sh \
> +				 test/scripts/makepkg-template_test.sh \
>  				 test/scripts/pacman-db-upgrade-v9.py \
>  				 test/util/pacsorttest.sh \
>  				 test/util/vercmptest.sh
> @@ -37,6 +38,7 @@ TEST_SUITE_LOG = test/test-suite.log
>  TEST_EXTENSIONS = .py
>  AM_TESTS_ENVIRONMENT = \
>  	PMTEST_UTIL_DIR=$(top_builddir)/src/util/; export PMTEST_UTIL_DIR; \
> +	PMTEST_SCRIPT_DIR=$(top_srcdir)/scripts/; export PMTEST_SCRIPT_DIR; \
>  	PMTEST_SCRIPTLIB_DIR=$(top_srcdir)/scripts/library/; export PMTEST_SCRIPTLIB_DIR;
>  LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
>  								 $(top_srcdir)/build-aux/tap-driver.sh
> diff --git a/test/scripts/Makefile.am b/test/scripts/Makefile.am
> index 8d6bc84..ed1a951 100644
> --- a/test/scripts/Makefile.am
> +++ b/test/scripts/Makefile.am
> @@ -1,6 +1,7 @@
>  check_SCRIPTS = \
>  	parseopts_test.sh \
>  	pacman-db-upgrade-v9.py \
> +	makepkg-template_test.sh \
>  	human_to_size_test.sh
>  
>  noinst_SCRIPTS = $(check_SCRIPTS)
> diff --git a/test/scripts/makepkg-template_test.sh b/test/scripts/makepkg-template_test.sh
> new file mode 100755
> index 0000000..5242f2c
> --- /dev/null
> +++ b/test/scripts/makepkg-template_test.sh
> @@ -0,0 +1,89 @@
> +#!/bin/bash
> +
> +declare -i testcount=0 fail=0 pass=0 total=0
> +
> +# source the library function
> +scriptdir=${1:-${PMTEST_SCRIPT_DIR}}
> +if [[ -z $scriptdir || ! -f $scriptdir/makepkg-template ]]; then
> +	printf "Bail out! makepkg-template executable (%s) could not be located\n" "${scriptdir}/makepkg-template"
> +	exit 1
> +fi

This does not work the same as our other tests which allow passing the
executable as the first argument.

> +
> +testdir="${0%/*}/makepkg-template-tests"
> +
> +total=$(find "$testdir" -maxdepth 1 -mindepth 1 -type d | wc -l)
> +
> +run_test() {
> +	local testcase=$1 exitcode
> +	local -i failed=0
> +
> +	(( ++testcount ))
> +
> +	local -a arguments
> +
> +	if [[ -f "$testdir/$testcase/arguments" ]]; then
> +		arguments=($(cat "$testdir/$testcase/arguments"))
> +	fi
> +
> +	mkdir "$TMPDIR/$testcase"
> +
> +	"$scriptdir/makepkg-template" \

makepkg-template uses translated messages; if you're going to check
its output you need to set the locale.

> +		--template-dir "$testdir/$testcase/templates" \
> +		-p "$testdir/$testcase/PKGBUILD" \
> +		-o "$TMPDIR/$testcase/result" \
> +		&> "$TMPDIR/$testcase/output_full" "${arguments[@]}"
> +	exitcode=$?
> +
> +	sed "$TMPDIR/$testcase/output_full" >"$TMPDIR/$testcase/output" \
> +		-e 's|./test/scripts/|./|' \

That should be $testdir or some variation of it.

> +		-e 's| at '$scriptdir'/makepkg-template line [0-9]\+\.$||'

makepkg-template should be updated to not include line numbers in
error messages rather than trying to filter them out.

> +	printf "exitcode: %s\n" "$exitcode" >> "$TMPDIR/$testcase/output"

It would be better if we could test the exit code, output, and result
independently.

> +
> +	if [[ -f "$TMPDIR/$testcase/result" ]]; then
> +		if ! diff -u "$TMPDIR/$testcase/result" "$testdir/$testcase/expected_result"; then

The diff output needs to be marked as diagnostic output.  I'm working
on a TAP generator for bash that we can use that will do that for us
as well as taking care of all the counting.

> +			failed=1
> +		fi
> +	else
> +		if [[ -f "$testdir/$testcase/expected_result" ]]; then
> +			echo "$testcase: Found expected_result file but no result file"
> +			failed=1
> +		fi
> +	fi
> +
> +	if ! diff -u "$TMPDIR/$testcase/output" "$testdir/$testcase/expected_output"; then
> +		failed=1
> +	fi
> +
> +	if ((!failed)); then
> +		(( ++pass ))
> +		printf "ok %d - %s\n" "$testcount" "$input"

I think you mean s/input/testcase/

> +	else
> +		(( ++fail ))
> +		printf "not ok %d - %s\n" "$testcount" "$input"
> +		printf '# [TEST %3s]: FAIL\n' "$testcount"

Redundant failure message.

> +	fi
> +}
> +
> +summarize() {
> +	rm -rf "$TMPDIR"
> +
> +	if (( !fail )); then
> +		printf '# All %s tests successful\n\n' "$testcount"
> +		exit 0
> +	else
> +		printf '# %s of %s tests failed\n\n' "$fail" "$testcount"
> +		exit 1
> +	fi
> +}
> +TMPDIR="$(mktemp -d "/tmp/${0##*/}.XXXXXX")"
> +trap 'summarize' EXIT
> +
> +printf '# Beginning makepkg-template tests\n'

I know we have these messages in other tests, but they are completely
useless, let's not add any more.

> +
> +echo "1..$total"
> +
> +for dir in "$testdir/"*; do
> +	if [[ -d "$dir" ]]; then
> +		run_test "${dir##*/}"
> +	fi
> +done
> -- 
> 2.2.1