- Aur-dev - lists.archlinux.org

[PATCH 1/2] Fix invalid session ID check
by Lukas Fleischer 05 Apr '20

05 Apr '20

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- web/lib/aur.inc.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/lib/aur.inc.php b/web/lib/aur.inc.php index dbcc23a..f4ad6b4 100644 --- a/web/lib/aur.inc.php +++ b/web/lib/aur.inc.php @@ -50,7 +50,7 @@ function check_sid() { $result = $dbh->query($q); $row = $result->fetch(PDO::FETCH_NUM); - if (!$row[0]) { + if (!$row) { # Invalid SessionID - hacker alert! # $failed = 1; -- 2.26.0

1 1

aurweb 5.0.0 released
by Lukas Fleischer 27 Mar '20

27 Mar '20

Dear aurweb contributors and users, aurweb 5.0.0 has just been released! You can now add a secondary email address that can be used to recover your account in case access to the primary email address is lost. Reset keys for an account are always sent to both the primary and the backup email address. Most other changes were internal. Shout-out to Frédéric Mangano-Tarumi for doing some amazing work on the backend! For a comprehensive list of changes, please consult the Git log [2]. As usual, bugs should be reported to the aurweb bug tracker [3]. [1] https://aur.archlinux.org/ [2] https://projects.archlinux.org/aurweb.git/log/?id=v5.0.0 [3] https://bugs.archlinux.org/index.php?project=2

1 0

[PATCH] Add new upgrade instructions
by Lukas Fleischer 27 Mar '20

27 Mar '20

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- upgrading/5.x.x.txt | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 upgrading/5.x.x.txt diff --git a/upgrading/5.x.x.txt b/upgrading/5.x.x.txt new file mode 100644 index 0000000..ba74e28 --- /dev/null +++ b/upgrading/5.x.x.txt @@ -0,0 +1,4 @@ +Starting from release 5.0.0, Alembic is used for managing database migrations. + +Run `alembic upgrade head` from the aurweb root directory to upgrade your +database after upgrading the source code to a new release. -- 2.26.0

1 1

TAP testing in Python
by Frédéric Mangano-Tarumi 23 Mar '20

23 Mar '20

In order to write tests in Python, we first need to make one important choice: the test framework. Our main constraint is the output format, which should be TAP to run along with the existing scripts. My favorite candidate is pycotap, which holds in a 157-line no-dependency module that we could put in our tests directory, pretty much like we did for `test/sharness.sh`. It is implemented as a producer for Python’s standard unittest module. https://el-tramo.be/pycotap/ Its main opponent is tappy, a set of tools for working with TAP. It integrates with unittest, pytest and nose. From the documentation, my impression is that it expects to be invoked with nose or pytest’s harness, which would be inconsistent with usual TAP test invocation consisting of calling scripts directly and relying on the shebang. https://tappy.readthedocs.io/en/latest/producers.html More alternatives at https://tappy.readthedocs.io/en/latest/alternatives.html Unless we plan to use pytest or nose over unittest, I suggest we use use pycotap. Except for a couple of lines, tests written with pycotap are standard unittest tests, which nose and pytest can collect results from too. If our needs evolve, we could easily upgrade to another framework.

3 4

[PATCH] Map BIGINT to INTEGER for SQLite
by Frédéric Mangano-Tarumi 22 Mar '20

22 Mar '20

--- aurweb/schema.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/aurweb/schema.py b/aurweb/schema.py index fde6512f..6792cf1d 100644 --- a/aurweb/schema.py +++ b/aurweb/schema.py @@ -17,6 +17,17 @@ def compile_tinyint_sqlite(type_, compiler, **kw): return 'INTEGER' +@compiles(BIGINT, 'sqlite') +def compile_bigint_sqlite(type_, compiler, **kw): + """ + For SQLite's AUTOINCREMENT to work on BIGINT columns, we need to map BIGINT + to INTEGER. Aside from that, BIGINT is the same as INTEGER for SQLite. + + See https://docs.sqlalchemy.org/en/13/dialects/sqlite.html#allowing-autoincreme… + """ + return 'INTEGER' + + metadata = MetaData() # Define the Account Types for the AUR. -- 2.25.2

2 1

[PATCH] Proof-of-concept of a dual PHP–Python stack
by Frédéric Mangano-Tarumi 15 Mar '20

15 Mar '20

Here’s a demonstration of a Python web stack running next to the current PHP stack, such that it’s invisible to the client. This approach aims at providing a way to migrate the PHP code base to Python, one endpoint after another, with as little glue as possible to have the two backends collaborate. Since they are both mounted on the web root, Python could implement, say, /packages/ and leave the rest to PHP. As soon as PHP yields it by returning 404 on this URL, Python will take over automatically. To run it, you need python-flask and nginx. Then, you need to start PHP, Flask, and nginx, in whatever order: $ cd path/to/aurweb $ AUR_CONFIG="$PWD/conf/config" php -S 127.0.0.1:8080 -t web/html $ FLASK_APP=aurweb.wsgi flask run $ nginx -p . -c conf/nginx.conf You may then open http://localhost:8081/ and http://localhost:8081/hello to check that the former URL goes to PHP and the latter to Flask. The key concept is nginx’s proxy_next_upstream feature. We set Flask as a fallback backend, and tell nginx to use the fallback only on 404 from PHP. See http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream The main limitation of this approach is that PHP and Python need to use the same gateway protocol, probably FastCGI or HTTP. A minor caveat with this system is that the body of the 404 returned by PHP is lost, though it could contain useful information like “Package X doesn’t exist”, rather than a generic “Page not found”. Luckily, all the 404 cases are handled by 404.php, so we could port its logic to Flask and preserve the current behavior. --- aurweb/wsgi.py | 15 +++++++++++++++ conf/nginx.conf | 23 +++++++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 aurweb/wsgi.py create mode 100644 conf/nginx.conf diff --git a/aurweb/wsgi.py b/aurweb/wsgi.py new file mode 100644 index 00000000..fd6b67d3 --- /dev/null +++ b/aurweb/wsgi.py @@ -0,0 +1,15 @@ +from flask import Flask, request + + +def create_app(): + app = Flask(__name__) + + @app.route('/hello', methods=['GET', 'POST']) + def hello(): + return ( + f"{request.method} {request.url}\n" + f"{request.headers}" + f"{request.get_data(as_text=True)}\n" + ), {'Content-Type': 'text/plain'} + + return app diff --git a/conf/nginx.conf b/conf/nginx.conf new file mode 100644 index 00000000..8e6e4edb --- /dev/null +++ b/conf/nginx.conf @@ -0,0 +1,23 @@ +events { +} + +daemon off; +error_log /dev/stderr info; +pid nginx.pid; + +http { + access_log /dev/stdout; + + upstream aurweb { + server [::1]:8080 max_fails=0; + server 127.0.0.1:5000 backup max_fails=0; + } + + server { + listen 8081; + location / { + proxy_pass http://aurweb; + proxy_next_upstream http_404 non_idempotent; + } + } +} -- 2.25.1

3 7

Functional HTTP testing
by Frédéric Mangano-Tarumi 15 Mar '20

15 Mar '20

Hello everyone, With the current plans of porting the PHP codebase to Python, we should consider setting up a suite of functional tests to squash most bugs introduced by the rewrite. I suggest the following testing plan, to be applied on an endpoint-by-endpoint basis: 1. Write functional HTTP tests for the existing software. 2. Port the code to Python. 3. Run the tests. This differs from the more appealing plan to test only the new code in Python using tools provided by the framework we’d use. I think testing the PHP code first will help find surprising edge cases we could accidentally change but that other parts of the system depend on. HTTP testing being a kind of language-agnostic blackbox testing, the tests would also resist any future rewrite or major refactoring without requiring maintenance. I’d now like to introduce a few solutions to implement that. Requests -------- Using Python’s Requests package with an HTML and a JSON parser should cover most cases. We would typically run POST requests to change the state then parse the HTML to validate the updated data. Any other programming language would work too. Perl offers better integration with our current test architecture, and its support for regex is quite practical for matching results. Mechanize --------- Perl has a cool WWW::Mechanize package that provides a higher-level interface to HTTP testing, offering basic support for forms and link detection. We would simulate use cases like “Click the ‘login’ link, submit the form, etc.”, ensuring the website is usable by a human user. More at https://metacpan.org/pod/WWW::Mechanize#SYNOPSIS Someone made a port of WWW::Mechanize for Python, called python-mechanize. Selenium -------- Selenium a software suite for testing web applications among other things. Its WebDriver component controls a full-fledged browser like Firefox, supporting in particular JavaScript. You can’t get more powerful than that. See https://www.selenium.dev/documentation/en/ I think it’s overkill because I believe AUR should never require JavaScript, as it should be usable without a GUI. I myself occasionally access it on a text browser. It’s also simpler for bots, including search engine’s. Conclusion ---------- I think it’s safe to start with Requests, and consider Mechanize later if need be, or even Selenium. Mechanize and Selenium kind of overlap with each other, so we should pick only one of the two, but Requests is well-known and should never be unwelcome. If it were only me, I’d pick Perl for its shell-like features and regex support that make it a great tool for writing tests in my opinion. However, not that many developers are familiar with Perl nowadays.

2 1

[PATCH 4/4] Write test/README.md to help working with tests
by Frédéric Mangano-Tarumi 29 Feb '20

29 Feb '20

--- test/README.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 test/README.md diff --git a/test/README.md b/test/README.md new file mode 100644 index 00000000..c48001d8 --- /dev/null +++ b/test/README.md @@ -0,0 +1,38 @@ +Running tests +------------- + +To run all the tests, you may run `make check` under `test/`. + +For more control, you may use the `prove` command, which receives a directory or +a list of files to run, and produces a report. + +Each test script is standalone, so you may run them individually. Some tests may +receive command-line options to help debugging. See for example sharness’s +documentation for shell test scripts: +https://github.com/chriscool/sharness/blob/master/README.git + +### Dependencies + +For all the test to run, the following Arch packages should be installed: + +- pyalpm +- python-alembic +- python-bleach +- python-markdown +- python-pygit2 +- python-sqlalchemy +- python-srcinfo + + +Writing tests +------------- + +Test scripts must follow the Test Anything Protocol specification: +http://testanything.org/tap-specification.html + +Tests must support being run from any directory. They may use $0 to determine +their location. Python scripts should expect aurweb to be installed and +importable without toying with os.path or PYTHONPATH. + +Tests written in shell should use sharness. In general, new tests should be +consistent with existing tests unless they have a good reason not to. -- 2.25.1

2 1

[PATCH 3/4] test/Makefile: Run tests with prove when available
by Frédéric Mangano-Tarumi 29 Feb '20

29 Feb '20

--- test/Makefile | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/test/Makefile b/test/Makefile index d310c8f5..758befa3 100644 --- a/test/Makefile +++ b/test/Makefile @@ -1,6 +1,13 @@ T = $(sort $(wildcard *.t)) +PROVE := $(shell command -v prove 2> /dev/null) + +ifdef PROVE +check: + prove . +else check: $(T) +endif clean: $(RM) -r test-results/ -- 2.25.1

2 1

[PATCH 2/4] Support running tests from any directory
by Frédéric Mangano-Tarumi 29 Feb '20

29 Feb '20

--- test/setup.sh | 5 ++--- test/t1100-git-auth.t | 2 +- test/t1200-git-serve.t | 2 +- test/t1300-git-update.t | 2 +- test/t2100-mkpkglists.t | 2 +- test/t2200-tuvotereminder.t | 2 +- test/t2300-pkgmaint.t | 2 +- test/t2400-aurblup.t | 2 +- test/t2500-notify.t | 2 +- test/t2600-rendercomment.t | 2 +- test/t2700-usermaint.t | 2 +- 11 files changed, 12 insertions(+), 13 deletions(-) diff --git a/test/setup.sh b/test/setup.sh index cad5cd66..4a6eb3b1 100644 --- a/test/setup.sh +++ b/test/setup.sh @@ -1,7 +1,6 @@ -TEST_DIRECTORY="$(pwd)" -TOPLEVEL="$(cd .. && pwd)" +TOPLEVEL="$(cd "$(dirname "$0")/.." && pwd)" -. ./sharness.sh +. "$TOPLEVEL/test/sharness.sh" # Configure python search path. PYTHONPATH="$TOPLEVEL" diff --git a/test/t1100-git-auth.t b/test/t1100-git-auth.t index 71d526f2..cbf16aed 100755 --- a/test/t1100-git-auth.t +++ b/test/t1100-git-auth.t @@ -2,7 +2,7 @@ test_description='git-auth tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test basic authentication.' ' "$GIT_AUTH" "$AUTH_KEYTYPE_USER" "$AUTH_KEYTEXT_USER" >out && diff --git a/test/t1200-git-serve.t b/test/t1200-git-serve.t index e817b2cf..1893cdcd 100755 --- a/test/t1200-git-serve.t +++ b/test/t1200-git-serve.t @@ -2,7 +2,7 @@ test_description='git-serve tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test interactive shell.' ' "$GIT_SERVE" 2>&1 | grep -q "Interactive shell is disabled." diff --git a/test/t1300-git-update.t b/test/t1300-git-update.t index 06d14984..82c0fb99 100755 --- a/test/t1300-git-update.t +++ b/test/t1300-git-update.t @@ -2,7 +2,7 @@ test_description='git-update tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" dump_package_info() { for t in Packages Licenses PackageLicenses Groups PackageGroups \ diff --git a/test/t2100-mkpkglists.t b/test/t2100-mkpkglists.t index fc11d073..5bf13de8 100755 --- a/test/t2100-mkpkglists.t +++ b/test/t2100-mkpkglists.t @@ -2,7 +2,7 @@ test_description='mkpkglists tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test package list generation with no packages.' ' echo "DELETE FROM Packages;" | sqlite3 aur.db && diff --git a/test/t2200-tuvotereminder.t b/test/t2200-tuvotereminder.t index c82ce874..5a8f3a25 100755 --- a/test/t2200-tuvotereminder.t +++ b/test/t2200-tuvotereminder.t @@ -2,7 +2,7 @@ test_description='tuvotereminder tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test Trusted User vote reminders.' ' now=$(date -d now +%s) && diff --git a/test/t2300-pkgmaint.t b/test/t2300-pkgmaint.t index 478df526..c390f5db 100755 --- a/test/t2300-pkgmaint.t +++ b/test/t2300-pkgmaint.t @@ -2,7 +2,7 @@ test_description='pkgmaint tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test package base cleanup script.' ' now=$(date -d now +%s) && diff --git a/test/t2400-aurblup.t b/test/t2400-aurblup.t index 708281c6..cc287a0f 100755 --- a/test/t2400-aurblup.t +++ b/test/t2400-aurblup.t @@ -2,7 +2,7 @@ test_description='aurblup tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test official provider update script.' ' mkdir -p remote/test/foobar-1.0-1 && diff --git a/test/t2500-notify.t b/test/t2500-notify.t index 380e65b8..5ef64c18 100755 --- a/test/t2500-notify.t +++ b/test/t2500-notify.t @@ -2,7 +2,7 @@ test_description='notify tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test out-of-date notifications.' ' cat <<-EOD | sqlite3 aur.db && diff --git a/test/t2600-rendercomment.t b/test/t2600-rendercomment.t index be408b80..e01904c6 100755 --- a/test/t2600-rendercomment.t +++ b/test/t2600-rendercomment.t @@ -2,7 +2,7 @@ test_description='rendercomment tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test comment rendering.' ' cat <<-EOD | sqlite3 aur.db && diff --git a/test/t2700-usermaint.t b/test/t2700-usermaint.t index 4f625142..f0bb449b 100755 --- a/test/t2700-usermaint.t +++ b/test/t2700-usermaint.t @@ -2,7 +2,7 @@ test_description='usermaint tests' -. ./setup.sh +. "$(dirname "$0")/setup.sh" test_expect_success 'Test removal of login IP addresses.' ' now=$(date -d now +%s) && -- 2.25.1

2 1