- Aur-dev - lists.archlinux.org

[aur-dev] 4.4.0 string freeze
by Lukas Fleischer 08 Oct '16

08 Oct '16

Dear translators, Strings for 4.4.0 have been submitted to Transifex. I plan to tag the release on Saturday, 2016-10-15, so there are ~7 days left to translate the new messages. Thank you for contributing! Regards, Lukas

1 0

[aur-dev] [PATCH] Update .gitignore
by Lukas Fleischer 08 Oct '16

08 Oct '16

Remove obsolete files. Add byte-compiled Python files and test results. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- .gitignore | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.gitignore b/.gitignore index c41bf3e..2cb5bdc 100644 --- a/.gitignore +++ b/.gitignore @@ -3,8 +3,8 @@ dummy-data.sql* po/*.mo po/*.po~ po/POTFILES -scripts/git-integration/__pycache__/ -scripts/git-integration/templates/ web/locale/*/ aur.git/ -git-interface/git-auth.sh +__pycache__/ +*.py[cod] +test/test-results/ -- 2.10.0

1 0

[aur-dev] [PATCH] Reorganize Git interface scripts
by Lukas Fleischer 08 Oct '16

08 Oct '16

Move the Git interface scripts from git-interface/ to aurweb/git/. Use setuptools to automatically create wrappers which can be installed using `python3 setup.py install`. Update the configuration files, the test suite as well as the INSTALL and README files to reflect these changes. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- INSTALL | 28 ++++++++++------------ README | 5 +--- git-interface/git-auth.py => aurweb/git/auth.py | 0 git-interface/git-serve.py => aurweb/git/serve.py | 0 .../git-update.py => aurweb/git/update.py | 0 conf/config.proto | 4 ++-- git-interface/Makefile | 18 -------------- git-interface/__init__.py | 0 git-interface/config.mk | 1 - git-interface/git-auth.sh.in | 3 --- setup.py | 7 ++++++ test/setup.sh | 8 +++---- 12 files changed, 26 insertions(+), 48 deletions(-) rename git-interface/git-auth.py => aurweb/git/auth.py (100%) rename git-interface/git-serve.py => aurweb/git/serve.py (100%) rename git-interface/git-update.py => aurweb/git/update.py (100%) delete mode 100644 git-interface/Makefile delete mode 100644 git-interface/__init__.py delete mode 100644 git-interface/config.mk delete mode 100644 git-interface/git-auth.sh.in diff --git a/INSTALL b/INSTALL index dab48cc..395915a 100644 --- a/INSTALL +++ b/INSTALL @@ -37,11 +37,16 @@ Setup on Arch Linux $ mysql -uaur -p AUR </srv/http/aurweb/schema/aur-schema.sql -5) Create a new user: +5) Install Python modules and dependencies: + + # pacman -S python-mysql-connector python-pygit2 python-srcinfo + # python3 setup.py install + +6) Create a new user: # useradd -U -d /srv/http/aurweb -c 'AUR user' aur -6) Initialize the Git repository: +7) Initialize the Git repository: # mkdir /srv/http/aurweb/aur.git/ # cd /srv/http/aurweb/aur.git/ @@ -49,29 +54,20 @@ Setup on Arch Linux # git config --local transfer.hideRefs '^refs/' # git config --local --add transfer.hideRefs '!refs/' # git config --local --add transfer.hideRefs '!HEAD' - # ln -s ../../git-interface/git-update.py hooks/update + # ln -s /usr/local/bin/aurweb-git-update hooks/update # chown -R aur . -7) Install needed Python modules: - - # pacman -S python-mysql-connector python-pygit2 python-srcinfo - -8) Install the git-auth wrapper script: - - # cd /srv/http/aurweb/git-interface/ - # make && make install - -9) Configure sshd(8) for the AUR. Add the following lines at the end of your +8) Configure sshd(8) for the AUR. Add the following lines at the end of your sshd_config(5) and restart the sshd. Note that OpenSSH 6.9 or newer is needed! Match User aur PasswordAuthentication no - AuthorizedKeysCommand /usr/local/bin/aur-git-auth "%t" "%k" + AuthorizedKeysCommand /usr/local/bin/aurweb-git-auth "%t" "%k" AuthorizedKeysCommandUser aur -10) If you want to enable smart HTTP support with nginx and fcgiwrap, you can - use the following directives: +9) If you want to enable smart HTTP support with nginx and fcgiwrap, you can + use the following directives: location ~ "^/([a-z0-9][a-z0-9.+_-]*?)(\.git)?/(git-(receive|upload)-pack|HEAD|info/refs|objects/(info/(http-)?alternates|packs)|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))$" { fastcgi_pass unix:/run/fcgiwrap.sock; diff --git a/README b/README index c110ef8..e633ec3 100644 --- a/README +++ b/README @@ -18,7 +18,7 @@ Directory Layout ---------------- aurweb:: - Shared aurweb Python modules. + aurweb Python modules. conf:: Configuration and configuration templates. @@ -26,9 +26,6 @@ conf:: doc:: Project documentation. -git-interface:: - Scripts that make up the Git and SSH interface. - po:: Translation files for strings in the aurweb interface. diff --git a/git-interface/git-auth.py b/aurweb/git/auth.py similarity index 100% rename from git-interface/git-auth.py rename to aurweb/git/auth.py diff --git a/git-interface/git-serve.py b/aurweb/git/serve.py similarity index 100% rename from git-interface/git-serve.py rename to aurweb/git/serve.py diff --git a/git-interface/git-update.py b/aurweb/git/update.py similarity index 100% rename from git-interface/git-update.py rename to aurweb/git/update.py diff --git a/conf/config.proto b/conf/config.proto index 21441a9..96fad80 100644 --- a/conf/config.proto +++ b/conf/config.proto @@ -46,14 +46,14 @@ RSA = SHA256:Ju+yWiMb/2O+gKQ9RJCDqvRg7l+Q95KFAeqM5sr6l2s [auth] valid-keytypes = ssh-rsa ssh-dss ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519 username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$ -git-serve-cmd = /srv/http/aurweb/git-interface/git-serve.py +git-serve-cmd = /usr/local/bin/aurweb-git-serve ssh-options = restrict [serve] repo-path = /srv/http/aurweb/aur.git/ repo-regex = [a-z0-9][a-z0-9.+_-]*$ git-shell-cmd = /usr/bin/git-shell -git-update-cmd = /srv/http/aurweb/git-interface/git-update.py +git-update-cmd = /usr/local/bin/aurweb-git-update ssh-cmdline = ssh aur(a)aur.archlinux.org [update] diff --git a/git-interface/Makefile b/git-interface/Makefile deleted file mode 100644 index 8865790..0000000 --- a/git-interface/Makefile +++ /dev/null @@ -1,18 +0,0 @@ -GIT_INTERFACE_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))) - -include config.mk - -git-auth.sh: - sed 's#%GIT_INTERFACE_DIR%#$(GIT_INTERFACE_DIR)#' <git-auth.sh.in >git-auth.sh - chmod +x git-auth.sh - -install: git-auth.sh - install -Dm0755 git-auth.sh "$(DESTDIR)$(PREFIX)/bin/aur-git-auth" - -uninstall: - rm -f "$(DESTDIR)$(PREFIX)/bin/aur-git-auth" - -clean: - rm -f git-auth.sh - -.PHONY: install uninstall clean diff --git a/git-interface/__init__.py b/git-interface/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/git-interface/config.mk b/git-interface/config.mk deleted file mode 100644 index 4d794a1..0000000 --- a/git-interface/config.mk +++ /dev/null @@ -1 +0,0 @@ -PREFIX = /usr/local diff --git a/git-interface/git-auth.sh.in b/git-interface/git-auth.sh.in deleted file mode 100644 index 223816a..0000000 --- a/git-interface/git-auth.sh.in +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/sh - -%GIT_INTERFACE_DIR%/git-auth.py "$1" "$2" diff --git a/setup.py b/setup.py index 48eb176..b64e71c 100644 --- a/setup.py +++ b/setup.py @@ -17,4 +17,11 @@ setup( name="aurweb", version=version, packages=find_packages(), + entry_points={ + 'console_scripts': [ + 'aurweb-git-auth = aurweb.git.auth:main', + 'aurweb-git-serve = aurweb.git.serve:main', + 'aurweb-git-update = aurweb.git.update:main', + ], + }, ) diff --git a/test/setup.sh b/test/setup.sh index dc9cff2..d02d298 100644 --- a/test/setup.sh +++ b/test/setup.sh @@ -8,9 +8,9 @@ PYTHONPATH="$TOPLEVEL" export PYTHONPATH # Configure paths to the Git interface scripts. -GIT_AUTH="$TOPLEVEL/git-interface/git-auth.py" -GIT_SERVE="$TOPLEVEL/git-interface/git-serve.py" -GIT_UPDATE="$TOPLEVEL/git-interface/git-update.py" +GIT_AUTH="$TOPLEVEL/aurweb/git/auth.py" +GIT_SERVE="$TOPLEVEL/aurweb/git/serve.py" +GIT_UPDATE="$TOPLEVEL/aurweb/git/update.py" MKPKGLISTS="$TOPLEVEL/scripts/mkpkglists.py" TUVOTEREMINDER="$TOPLEVEL/scripts/tuvotereminder.py" PKGMAINT="$TOPLEVEL/scripts/pkgmaint.py" @@ -38,7 +38,7 @@ reply-to = noreply(a)aur.archlinux.org [auth] valid-keytypes = ssh-rsa ssh-dss ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519 username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$ -git-serve-cmd = /srv/http/aurweb/git-interface/git-serve.py +git-serve-cmd = $GIT_SERVE ssh-options = restrict [serve] -- 2.10.0

1 0

[aur-dev] [PATCH] Add clone hints to 404 error pages
by Lukas Fleischer 08 Oct '16

08 Oct '16

When clicking on the linked Git clone URL of a package base, users are faced with a 404 error page since the URL is not supposed to be opened in a web browser. Add some notes to 404 error pages corresponding to Git clone URLs that explain how to use them instead. Fixes FS#51266. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- web/html/404.php | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/web/html/404.php b/web/html/404.php index a47ae1e..757c485 100644 --- a/web/html/404.php +++ b/web/html/404.php @@ -2,12 +2,45 @@ set_include_path(get_include_path() . PATH_SEPARATOR . '../lib'); +$path = $_SERVER['PATH_INFO']; +$tokens = explode('/', $path); + +if (preg_match('/^([a-z0-9][a-z0-9.+_-]*?)(\.git)?$/', $tokens[1], $matches)) { + $gitpkg = $matches[1]; + if (pkg_from_name($gitpkg)) { + $gitcmd = 'git clone ' . sprintf(config_get('options', 'git_clone_uri_anon'), htmlspecialchars($gitpkg)); + $gitlink = get_pkgbase_uri($gitpkg); + } else { + unset($gitpkg); + } +} else { + unset($gitpkg); +} + html_header( __("Page Not Found") ); ?> <div id="error-page" class="box 404"> <h2>404 - <?= __("Page Not Found") ?></h2> <p><?= __("Sorry, the page you've requested does not exist.") ?></p> + <?php if (isset($gitpkg)): ?> + <ul> + <li> + <strong><?= __("Note") ?>:</strong> + <?= __("Git clone URLs are not meant to be opened in a browser.") ?> + </li> + <li> + <?= __("To clone the Git repository of %s, run %s.", + '<strong>' . htmlspecialchars($gitpkg) . '</strong>', + '<code>' . htmlspecialchars($gitcmd) . '</code>') ?> + </li> + <li> + <?= __("Click %shere%s to return to the %s details page.", + '<a href="' . htmlspecialchars($gitlink, ENT_QUOTES) . '">', '</a>', + '<strong>' . htmlspecialchars($gitpkg) . '</strong>') ?> + </li> + </ul> + <?php endif; ?> </div> <?php -- 2.10.0

1 0

[aur-dev] [PATCH] git-serve: Support `git {receive,upload}-pack`
by Lukas Fleischer 01 Oct '16

01 Oct '16

Add support for the `git receive-pack` and `git upload-pack` commands which are aliases for git-receive-pack and git-upload-pack, respectively. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-serve.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/git-interface/git-serve.py b/git-interface/git-serve.py index 5f3b26d..ebfef94 100755 --- a/git-interface/git-serve.py +++ b/git-interface/git-serve.py @@ -305,6 +305,10 @@ def main(): if remote_addr not in maintenance_exc: die("The AUR is down due to maintenance. We will be back soon.") + if action == 'git' and cmdargv[1] in ('upload-pack', 'receive-pack'): + action = action + '-' + cmdargv[1] + del cmdargv[1] + if action == 'git-upload-pack' or action == 'git-receive-pack': if len(cmdargv) < 2: die_with_help("{:s}: missing path".format(action)) -- 2.10.0

1 0

[aur-dev] Absence of terminating newline in RPC requests
by Alad Wenter 01 Oct '16

01 Oct '16

JSON from the RPC interface seems to not contain a terminating newline, regardless of version or type specified. I am curious if there is a particular reason for this, as the POSIX standard defines text files to contain lines with terminating newlines. [1] [2] This is more noticeable when merging the resulting files to a valid JSON structure, e.g. to display the union of various search results. As a comparison, journalctl - json does terminate lines with a newline. Regards, Alad [1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag… [2] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag…

2 3

[aur-dev] [PATCH 1/3] tuvotereminder: Wait for notify processes
by Lukas Fleischer 29 Sep '16

29 Sep '16

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- scripts/tuvotereminder.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/tuvotereminder.py b/scripts/tuvotereminder.py index a053608..97b1d12 100755 --- a/scripts/tuvotereminder.py +++ b/scripts/tuvotereminder.py @@ -21,7 +21,7 @@ def main(): [filter_from, filter_to]) for vote_id in [row[0] for row in cur.fetchall()]: - subprocess.Popen((notify_cmd, 'tu-vote-reminder', str(vote_id))) + subprocess.Popen((notify_cmd, 'tu-vote-reminder', str(vote_id))).wait() if __name__ == '__main__': -- 2.10.0

1 2

[aur-dev] [PATCH/RFC] Send reminders before the TU voting period ends
by Lukas Fleischer 29 Sep '16

29 Sep '16

Add a new script that checks for TU votes ending within the next 48 hours and sends reminder emails to all Trusted Users who did not cast their votes yet. The script is designed to be executed as a cron job, such that the check is performed periodically. Since the script does not remember users it already sent emails to, the interval should not be chosen too small to avoid spamming Trusted Users. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- I did not test this at all. Seems like a good idea to reduce the number of skipped votes, though. scripts/notify.py | 23 +++++++++++++++++++++++ scripts/tuvotereminder.py | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100755 scripts/tuvotereminder.py diff --git a/scripts/notify.py b/scripts/notify.py index 6ea25d1..aeb59c3 100755 --- a/scripts/notify.py +++ b/scripts/notify.py @@ -139,6 +139,15 @@ def get_request_recipients(cur, reqid): return [row[0] for row in cur.fetchall()] +def get_tu_vote_reminder_recipients(cur, vote_id): + cur.execute('SELECT ID FROM Users WHERE AccountTypeID = 2 ' + + 'EXCEPT SELECT Users.ID FROM Users ' + + 'INNER JOIN TU_Votes ' + + 'ON TU_Votes.UserID = Users.ID ' + + 'WHERE TU_Votes.VoteID = %s', [vote_id]) + return [row[0] for row in cur.fetchall()] + + def get_comment(cur, comment_id): cur.execute('SELECT Comments FROM PackageComments WHERE ID = %s', [comment_id]) @@ -405,6 +414,19 @@ def request_close(cur, uid, reqid, reason): send_notification(to, subject, body, refs, headers) +def tu_vote_reminder(cur, vote_id): + to = get_tu_vote_reminder_recipients(cur, vote_id) + + vote_uri = aur_location + '/tu/?id=' + vote_id + + subject = 'TU Vote Reminder: Proposal %d' % (vote_id) + body = 'Please remember to cast your vote on proposal %d [1]. ' \ + 'The voting period ends in less than 48 hours.' + refs = '[1] ' + vote_uri + + send_notification(to, subject, body, refs) + + if __name__ == '__main__': action = sys.argv[1] action_map = { @@ -420,6 +442,7 @@ if __name__ == '__main__': 'delete': delete, 'request-open': request_open, 'request-close': request_close, + 'tu-vote-reminder': tu_vote_reminder, } db = mysql.connector.connect(host=aur_db_host, user=aur_db_user, diff --git a/scripts/tuvotereminder.py b/scripts/tuvotereminder.py new file mode 100755 index 0000000..0992623 --- /dev/null +++ b/scripts/tuvotereminder.py @@ -0,0 +1,32 @@ +#!/usr/bin/python3 + +import configparser +import mysql.connector +import os +import subprocess +import time + +config = configparser.RawConfigParser() +config.read(os.path.dirname(os.path.realpath(__file__)) + "/../conf/config") + +aur_db_host = config.get('database', 'host') +aur_db_name = config.get('database', 'name') +aur_db_user = config.get('database', 'user') +aur_db_pass = config.get('database', 'password') +aur_db_socket = config.get('database', 'socket') +notify_cmd = config.get('notifications', 'notify-cmd') + +db = mysql.connector.connect(host=aur_db_host, user=aur_db_user, + passwd=aur_db_pass, db=aur_db_name, + unix_socket=aur_db_socket, buffered=True) +cur = db.cursor() + +now = int(time.time()) +filter_from = now + 500 +filter_to = now + 172800 + +cur.execute("SELECT ID FROM TU_VoteInfo WHERE End >= %s AND End <= %s", + [filter_from, filter_to]) + +for vote_id in [row[0] for row in cur.fetchall()]: + subprocess.Popen((notify_cmd, 'tu-vote-reminder', str(vote_id))) -- 2.9.3

1 1

[aur-dev] Code complexity survey
by Vard Antinyan 27 Sep '16

27 Sep '16

Dear developers, We have undertaken a task to assess code complexity triggers and generate recommendations for developing understandable code. Our intension is to share the results with you, developers, so everyone can learn the triggers behind complex software. We need your help for rigorous results. My request to you is - if you get 5-10 min. time, would you please consider to answer the questions of this survey on code complexity? https://goo.gl/forms/h9WXZ8VSEw7BUyHg1 You are welcome to learn preliminary results through this link: https://www.facebook.com/SoftwareCodeQuality/photos/?tab=album&album_id=163… The results will be shared in a public webpage and everyone possible will be invited to learn and discuss them. Your knowledge and experience is vital for achieving substantial and generalizable results, and your effort is much appreciated! Sincerely Vard Antinyan PhD candidate in University of Gothenburg, Sweden Tel: 0046317725707

1 0

[aur-dev] [PATCH 00/17] Add tests for Python scripts
by Lukas Fleischer 26 Sep '16

26 Sep '16

Various changes in the way Python scripts and tests are organized. Shared Python modules are now placed in a site-packages directory. Tests for mkpkglists, tuvotereminder, pkgmaint and aurblup are added. Lukas Fleischer (17): Add a main() method to all Python scripts git-auth: Move entry point to a main() method git-serve: Pass user and privileges as parameters git-serve: Move entry point to a main() method git-update: Move entry point to a main() method Use setuptools to install Python modules Use config and db in scripts Make test suite paths top-level directory relative Reorganize tests Update README mkpkglists: Make output files configurable Add tests for mkpkglists Add tests for tuvotereminder scripts: Do not use UNIX_TIMESTAMP Add tests for pkgmaint aurblup: Drop support for multiple servers Add tests for aurblup README | 6 + aurweb/__init__.py | 0 {git-interface => aurweb}/config.py | 0 {git-interface => aurweb}/db.py | 16 +- conf/config.proto | 6 +- git-interface/__init__.py | 0 git-interface/git-auth.py | 58 ++-- git-interface/git-serve.py | 270 ++++++++++--------- git-interface/git-update.py | 300 +++++++++++---------- scripts/__init__.py | 0 scripts/aurblup.py | 105 ++++---- scripts/mkpkglists.py | 71 +++-- scripts/notify.py | 278 ++++++++++--------- scripts/pkgmaint.py | 31 +-- scripts/popupdate.py | 52 ++-- scripts/tuvotereminder.py | 40 ++- setup.py | 20 ++ {git-interface/test => test}/Makefile | 0 {git-interface/test => test}/setup.sh | 29 +- {git-interface/test => test}/sharness.sh | 0 .../test/t0001-auth.sh => test/t1100-git-auth.sh | 0 .../test/t0002-serve.sh => test/t1200-git-serve.sh | 0 .../t0003-update.sh => test/t1300-git-update.sh | 0 test/t2100-mkpkglists.sh | 47 ++++ test/t2200-tuvotereminder.sh | 26 ++ test/t2300-pkgmaint.sh | 26 ++ test/t2400-aurblup.sh | 53 ++++ 27 files changed, 818 insertions(+), 616 deletions(-) create mode 100644 aurweb/__init__.py rename {git-interface => aurweb}/config.py (100%) rename {git-interface => aurweb}/db.py (73%) create mode 100644 git-interface/__init__.py create mode 100644 scripts/__init__.py create mode 100644 setup.py rename {git-interface/test => test}/Makefile (100%) rename {git-interface/test => test}/setup.sh (89%) rename {git-interface/test => test}/sharness.sh (100%) rename git-interface/test/t0001-auth.sh => test/t1100-git-auth.sh (100%) rename git-interface/test/t0002-serve.sh => test/t1200-git-serve.sh (100%) rename git-interface/test/t0003-update.sh => test/t1300-git-update.sh (100%) create mode 100755 test/t2100-mkpkglists.sh create mode 100755 test/t2200-tuvotereminder.sh create mode 100755 test/t2300-pkgmaint.sh create mode 100755 test/t2400-aurblup.sh -- 2.10.0

1 17