- Aur-dev - lists.archlinux.org

TAP conventions for aurweb’s test suites
by Frédéric Mangano-Tarumi 22 Feb '20

22 Feb '20

Hello everyone, The aurweb test suites are are currently run by make, looking for all files matching `tNNNN-*.sh`. See `test\Makefile`. Each test program is standalone and follows the TAP conventions. More about TAP at https://en.wikipedia.org/wiki/Test_Anything_Protocol While make is better than nothing, every Arch Linux system comes with a very common TAP harness: prove, shipped with Perl. The prove command line is the most intuitive it can get, and generates reports, keeps running tests even if one fails (unlike make which stops at the first failure), and can even run tests in parallel provided the tests support it. We can already use it by running `prove t*.sh` in the `test` directory. prove comes with its conventions, which are from the original TAP. First, it expects test programs to end with `.t`, which is restrictive enough to exclude non-test files, but doesn’t bind to any specific language since the shebang is used to know how to run each test. This is by the way a great opportunity to start writing tests in Python too. Another convention is to name the test directory `t` and to call prove from the root of the project. This one is harder to use since all the existing tests assume they’re run from the `test` directory. A simple cd at the beginning of every script is probably a good solution, but let’s leave that problem open for now. While we can run prove from the `test` directory, running tests from the root has the notable advantage of letting us import aurweb Python modules without hacking PYTHONPATH. Here are various suggestions: 1. Change the extension of all the test programs to `.t`. 2. Create `test\README` to tell developers about TAP and prove. 3. Make `make check` call prove. 4. Delete the `make t1234-xxx.sh` rules. 5. Delete the whole Makefile altogether. 6. Run the test programs from the project root. 7. Rename the `test` directory to `t`. 1 to 4 are cheap and have a significant added value in my opinion. 5 to 7 less so, but I think that if we want to use TAP, we should stick to its conventions. Feedback welcome! Cheers, -- fmang

2 3

[PATCH] Fix HTML code in the account search results table
by Lukas Fleischer 22 Feb '20

22 Feb '20

Do not add an opening <tbody> tag for every row. Instead, wrap all rows in <tbody></tbody>. While at it, also simplify the code used to color the rows. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- web/template/account_search_results.php | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/web/template/account_search_results.php b/web/template/account_search_results.php index 81cd818..0f7eb7a 100644 --- a/web/template/account_search_results.php +++ b/web/template/account_search_results.php @@ -16,17 +16,9 @@ else: <th><?= __("Edit Account") ?></th> </tr> </thead> - <?php - $i = 0; - foreach ($userinfo as $indx => $row): - if ($i % 2): - $c = "even"; - else: - $c = "odd"; - endif; - ?> - <tbody> - <tr class ="<?= $c ?>"> + <tbody> + <?php foreach ($userinfo as $idx => $row): ?> + <tr class ="<?= ($idx % 2 == 0) ? 'odd' : 'even' ?>"> <td><a href="<?= get_uri('/packages/'); ?>?SeB=m&K=<?= $row["Username"] ?>"><?= $row["Username"] ?></a></td> <td><?= $row["AccountType"] ?></td> <td> @@ -49,10 +41,8 @@ else: <?php endif; ?> </td> </tr> - <?php - $i++; - endforeach; - ?> + <?php endforeach; ?> + </tbody> </table> <table class="results"> -- 2.25.1

1 0

[PATCH 1/2] README.md: fix a small typo
by Lukas Fleischer 21 Feb '20

21 Feb '20

From: Yaron Shahrabani <sh.yaron(a)gmail.com> Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a4ab584..b2095b3 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ and installed using the Arch Linux package manager `pacman`. The aurweb project includes * A web interface to search for packaging scripts and display package details. -* A SSH/Git interface to submit and update packages and package meta data. +* An SSH/Git interface to submit and update packages and package meta data. * Community features such as comments, votes, package flagging and requests. * Editing/deletion of packages and accounts by Trusted Users and Developers. * Area for Trusted Users to post AUR-related proposals and vote on them. -- 2.25.1

1 1

Automated database migrations with Alembic
by Frédéric Mangano-Tarumi 15 Feb '20

15 Feb '20

Hello everyone, After discussing with Lukas about automation and database migrations, he suggested I write a proposal. Here it is. By the way, I’m a new contributor, and nothing has yet been decided on who would implement this proposal if we do decide to follow it. Feedback and contributions welcome! Context ======= We want to be able to deploy new versions of aurweb from Ansible without requiring manual intervention for data migration. Many all-in-one web frameworks provide tools for database migrations, but aurweb being a multi-language project, we can’t afford to make a switch to such a framework in the short term. Proposed solution: Alembic ========================== Disclaimer: I don’t have any practical experience with SQLAlchemy or Alembic in production. I’m suggesting it entirely because it looks like the best fit for our use case among the existing tools, but I could be missing something. Alembic[1] is a tool for managing database migrations. It is a sister project of the Python database library SQLAlchemy. Both Alembic and SQLAlchemy are flexible enough to collaborate with other database connectors. I suggest we use SQLAlchemy only for the migrations and the schema definition, without porting the existing Python and PHP codebase for now. The migrations scripts are written in Python and are intended to be used with SQLAlchemy, but that’s not a requirement. Regardless, I do suggest we migrate the whole schema to SQLAlchemy because (a) it takes care of the SQLite/MySQL compatibility for us, (b) Alembic is able to automatically generate migrations from SQLAlchemy schemas[2], and (c) we did plan to do that eventually, didn’t we? Alembic supports multi-database setups but don’t technically support migrating other data files like INI configurations, though it is something we might need to migrate in the future. We can still write custom code in the migrations, but with the risk that the files may get out of sync with their migration metadata in the database, which may happen after manual interventions. I think it’s worth it. To convert our current SQL schema, sqlacodegen[3] can generate the SQLAlchemy code for us. Alembic can also create migrations to migrate from zero to the full database, but I suggest we keep the SQLAlchemy schema up-to-date in the code. Migrations would only run on existing database, as described in the Alembic cookbook[4]. [1]: https://alembic.sqlalchemy.org/en/latest/tutorial.html [2]: https://alembic.sqlalchemy.org/en/latest/autogenerate.html [3]: https://pypi.org/project/sqlacodegen/ [4]: https://alembic.sqlalchemy.org/en/latest/cookbook.html#building-an-up-to-da… Alternative solutions ===================== Flyway and Liquibase are two popular tools for database version control. They’re both commercial with opensource editions, and written in Java. While not bad in themselves, Alembic is a better match for us with its licensing and its Python ecosystem. We could roll our own migration tools, but it would certainly require more effort than using an existing tool, and error handling is tricky and may cause data damage if not done right.

2 2

[aur-dev][PATCH 0/2] php warning fixes
by Eli Schwartz 12 Feb '20

12 Feb '20

I had this sitting around since December and forgot to send it in, offered FWIW. It's slightly different in implementation and has more wordy commit messages. Eli Schwartz (2): fix php 7.4 warnings fix more php 7.4 warnings web/lib/acctfuncs.inc.php | 1 + web/lib/aur.inc.php | 28 +++++++++++++++++++++------- web/lib/pkgfuncs.inc.php | 4 +++- 3 files changed, 25 insertions(+), 8 deletions(-) -- 2.25.0

2 3

[PATCH] Verify that return rows exist before extracting columns
by Lukas Fleischer 12 Feb '20

12 Feb '20

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- web/lib/aur.inc.php | 21 +++++++++++++++++++++ web/lib/pkgfuncs.inc.php | 3 +++ 2 files changed, 24 insertions(+) diff --git a/web/lib/aur.inc.php b/web/lib/aur.inc.php index e9530fc..2507df6 100644 --- a/web/lib/aur.inc.php +++ b/web/lib/aur.inc.php @@ -197,6 +197,9 @@ function username_from_id($id) { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } @@ -222,6 +225,9 @@ function username_from_sid($sid="") { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } @@ -339,6 +345,9 @@ function email_from_sid($sid="") { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } @@ -365,6 +374,9 @@ function account_from_sid($sid="") { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } @@ -390,6 +402,9 @@ function uid_from_sid($sid="") { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } @@ -512,6 +527,9 @@ function uid_from_username($username) { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } @@ -546,6 +564,9 @@ function uid_from_email($email) { } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } diff --git a/web/lib/pkgfuncs.inc.php b/web/lib/pkgfuncs.inc.php index a4cd17a..b30bfa9 100644 --- a/web/lib/pkgfuncs.inc.php +++ b/web/lib/pkgfuncs.inc.php @@ -147,6 +147,9 @@ function pkg_from_name($name="") { return; } $row = $result->fetch(PDO::FETCH_NUM); + if (!$row) { + return null; + } return $row[0]; } -- 2.25.0

1 1

[PATCH] Use relative URIs for {source_file,log,commit}_uri
by Lukas Fleischer 11 Feb '20

11 Feb '20

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- conf/config.defaults | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/conf/config.defaults b/conf/config.defaults index b69d031..447daca 100644 --- a/conf/config.defaults +++ b/conf/config.defaults @@ -28,9 +28,9 @@ request_idle_time = 1209600 request_archive_time = 15552000 auto_orphan_age = 15552000 auto_delete_age = 86400 -source_file_uri = https://aur.archlinux.org/cgit/aur.git/tree/%s?h=%s -log_uri = https://aur.archlinux.org/cgit/aur.git/log/?h=%s -commit_uri = https://aur.archlinux.org/cgit/aur.git/commit/?h=%s&id=%s +source_file_uri = /cgit/aur.git/tree/%s?h=%s +log_uri = /cgit/aur.git/log/?h=%s +commit_uri = /cgit/aur.git/commit/?h=%s&id=%s snapshot_uri = /cgit/aur.git/snapshot/%s.tar.gz enable-maintenance = 1 maintenance-exceptions = 127.0.0.1 -- 2.25.0

1 0

[PATCH v5 1/2] Support smtplib for sending emails
by Lukas Fleischer 11 Feb '20

11 Feb '20

Support mail delivery without a local MTA. Instead, an SMTP server can now be configured using the smtp-server option in the [notifications] section. In order to use this option, the value of the sendmail option must be empty. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- Disable debug output. aurweb/scripts/notify.py | 22 ++++++++++++++++++---- conf/config.defaults | 3 ++- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/aurweb/scripts/notify.py b/aurweb/scripts/notify.py index b0f218b..6c5c709 100755 --- a/aurweb/scripts/notify.py +++ b/aurweb/scripts/notify.py @@ -1,6 +1,8 @@ #!/usr/bin/env python3 import email.mime.text +import email.utils +import smtplib import subprocess import sys import textwrap @@ -63,7 +65,6 @@ class Notification: return body.rstrip() def send(self): - sendmail = aurweb.config.get('notifications', 'sendmail') sender = aurweb.config.get('notifications', 'sender') reply_to = aurweb.config.get('notifications', 'reply-to') reason = self.__class__.__name__ @@ -79,13 +80,26 @@ class Notification: msg['Reply-to'] = reply_to msg['To'] = to msg['X-AUR-Reason'] = reason + msg['Date'] = email.utils.formatdate(localtime=True) for key, value in self.get_headers().items(): msg[key] = value - p = subprocess.Popen([sendmail, '-t', '-oi'], - stdin=subprocess.PIPE) - p.communicate(msg.as_bytes()) + sendmail = aurweb.config.get('notifications', 'sendmail') + if sendmail: + # send email using the sendmail binary specified in the + # configuration file + p = subprocess.Popen([sendmail, '-t', '-oi'], + stdin=subprocess.PIPE) + p.communicate(msg.as_bytes()) + else: + # send email using smtplib; no local MTA required + server_addr = aurweb.config.get('notifications', 'smtp-server') + + server = smtplib.SMTP(server_addr) + server.set_debuglevel(0) + server.sendmail(sender, recipient, msg.as_bytes()) + server.quit() class ResetKeyNotification(Notification): diff --git a/conf/config.defaults b/conf/config.defaults index c519eae..23d46b0 100644 --- a/conf/config.defaults +++ b/conf/config.defaults @@ -47,7 +47,8 @@ window_length = 86400 [notifications] notify-cmd = /usr/local/bin/aurweb-notify -sendmail = /usr/bin/sendmail +sendmail = +smtp-server = localhost sender = notify(a)aur.archlinux.org reply-to = noreply(a)aur.archlinux.org -- 2.25.0

1 1

[PATCH v4 1/2] Support smtplib for sending emails
by Lukas Fleischer 10 Feb '20

10 Feb '20

Support mail delivery without a local MTA. Instead, an SMTP server can now be configured using the smtp-server option in the [notifications] section. In order to use this option, the value of the sendmail option must be empty. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- aurweb/scripts/notify.py | 22 ++++++++++++++++++---- conf/config.defaults | 3 ++- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/aurweb/scripts/notify.py b/aurweb/scripts/notify.py index b0f218b..941187e 100755 --- a/aurweb/scripts/notify.py +++ b/aurweb/scripts/notify.py @@ -1,6 +1,8 @@ #!/usr/bin/env python3 import email.mime.text +import email.utils +import smtplib import subprocess import sys import textwrap @@ -63,7 +65,6 @@ class Notification: return body.rstrip() def send(self): - sendmail = aurweb.config.get('notifications', 'sendmail') sender = aurweb.config.get('notifications', 'sender') reply_to = aurweb.config.get('notifications', 'reply-to') reason = self.__class__.__name__ @@ -79,13 +80,26 @@ class Notification: msg['Reply-to'] = reply_to msg['To'] = to msg['X-AUR-Reason'] = reason + msg['Date'] = email.utils.formatdate(localtime=True) for key, value in self.get_headers().items(): msg[key] = value - p = subprocess.Popen([sendmail, '-t', '-oi'], - stdin=subprocess.PIPE) - p.communicate(msg.as_bytes()) + sendmail = aurweb.config.get('notifications', 'sendmail') + if sendmail: + # send email using the sendmail binary specified in the + # configuration file + p = subprocess.Popen([sendmail, '-t', '-oi'], + stdin=subprocess.PIPE) + p.communicate(msg.as_bytes()) + else: + # send email using smtplib; no local MTA required + server_addr = aurweb.config.get('notifications', 'smtp-server') + + server = smtplib.SMTP(server_addr) + server.set_debuglevel(1) + server.sendmail(sender, recipient, msg.as_bytes()) + server.quit() class ResetKeyNotification(Notification): diff --git a/conf/config.defaults b/conf/config.defaults index c519eae..23d46b0 100644 --- a/conf/config.defaults +++ b/conf/config.defaults @@ -47,7 +47,8 @@ window_length = 86400 [notifications] notify-cmd = /usr/local/bin/aurweb-notify -sendmail = /usr/bin/sendmail +sendmail = +smtp-server = localhost sender = notify(a)aur.archlinux.org reply-to = noreply(a)aur.archlinux.org -- 2.25.0

1 1

[PATCH] Update README and convert to Markdown syntax
by Lukas Fleischer 10 Feb '20

10 Feb '20

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- README => README.md | 34 ++++++++-------------------------- 1 file changed, 8 insertions(+), 26 deletions(-) rename README => README.md (73%) diff --git a/README b/README.md similarity index 73% rename from README rename to README.md index e633ec3..a4ab584 100644 --- a/README +++ b/README.md @@ -17,32 +17,14 @@ The aurweb project includes Directory Layout ---------------- -aurweb:: - aurweb Python modules. - -conf:: - Configuration and configuration templates. - -doc:: - Project documentation. - -po:: - Translation files for strings in the aurweb interface. - -schema:: - Schema for the SQL database. Script for dummy data generation. - -scripts:: - Scripts for AUR maintenance. - -test:: - Test suite and test cases. - -upgrading:: - Instructions for upgrading setups from one release to another. - -web:: - Web interface for the AUR. +* `aurweb`: aurweb Python modules, Git interface and maintenance scripts +* `conf`: configuration and configuration templates +* `doc`: project documentation +* `po`: translation files for strings in the aurweb interface +* `schema`: schema for the SQL database +* `test`: test suite and test cases +* `upgrading`: instructions for upgrading setups from one release to another +* `web`: web interface for the AUR Links ----- -- 2.25.0

1 0