September 2016 - Aur-dev - lists.archlinux.org

[aur-dev] Absence of terminating newline in RPC requests
by Alad Wenter 01 Oct '16

01 Oct '16

JSON from the RPC interface seems to not contain a terminating newline, regardless of version or type specified. I am curious if there is a particular reason for this, as the POSIX standard defines text files to contain lines with terminating newlines. [1] [2] This is more noticeable when merging the resulting files to a valid JSON structure, e.g. to display the union of various search results. As a comparison, journalctl - json does terminate lines with a newline. Regards, Alad [1] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag… [2] http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag…

2 3

[aur-dev] [PATCH 1/3] tuvotereminder: Wait for notify processes
by Lukas Fleischer 29 Sep '16

29 Sep '16

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- scripts/tuvotereminder.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/tuvotereminder.py b/scripts/tuvotereminder.py index a053608..97b1d12 100755 --- a/scripts/tuvotereminder.py +++ b/scripts/tuvotereminder.py @@ -21,7 +21,7 @@ def main(): [filter_from, filter_to]) for vote_id in [row[0] for row in cur.fetchall()]: - subprocess.Popen((notify_cmd, 'tu-vote-reminder', str(vote_id))) + subprocess.Popen((notify_cmd, 'tu-vote-reminder', str(vote_id))).wait() if __name__ == '__main__': -- 2.10.0

1 2

[aur-dev] [PATCH/RFC] Send reminders before the TU voting period ends
by Lukas Fleischer 29 Sep '16

29 Sep '16

Add a new script that checks for TU votes ending within the next 48 hours and sends reminder emails to all Trusted Users who did not cast their votes yet. The script is designed to be executed as a cron job, such that the check is performed periodically. Since the script does not remember users it already sent emails to, the interval should not be chosen too small to avoid spamming Trusted Users. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- I did not test this at all. Seems like a good idea to reduce the number of skipped votes, though. scripts/notify.py | 23 +++++++++++++++++++++++ scripts/tuvotereminder.py | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100755 scripts/tuvotereminder.py diff --git a/scripts/notify.py b/scripts/notify.py index 6ea25d1..aeb59c3 100755 --- a/scripts/notify.py +++ b/scripts/notify.py @@ -139,6 +139,15 @@ def get_request_recipients(cur, reqid): return [row[0] for row in cur.fetchall()] +def get_tu_vote_reminder_recipients(cur, vote_id): + cur.execute('SELECT ID FROM Users WHERE AccountTypeID = 2 ' + + 'EXCEPT SELECT Users.ID FROM Users ' + + 'INNER JOIN TU_Votes ' + + 'ON TU_Votes.UserID = Users.ID ' + + 'WHERE TU_Votes.VoteID = %s', [vote_id]) + return [row[0] for row in cur.fetchall()] + + def get_comment(cur, comment_id): cur.execute('SELECT Comments FROM PackageComments WHERE ID = %s', [comment_id]) @@ -405,6 +414,19 @@ def request_close(cur, uid, reqid, reason): send_notification(to, subject, body, refs, headers) +def tu_vote_reminder(cur, vote_id): + to = get_tu_vote_reminder_recipients(cur, vote_id) + + vote_uri = aur_location + '/tu/?id=' + vote_id + + subject = 'TU Vote Reminder: Proposal %d' % (vote_id) + body = 'Please remember to cast your vote on proposal %d [1]. ' \ + 'The voting period ends in less than 48 hours.' + refs = '[1] ' + vote_uri + + send_notification(to, subject, body, refs) + + if __name__ == '__main__': action = sys.argv[1] action_map = { @@ -420,6 +442,7 @@ if __name__ == '__main__': 'delete': delete, 'request-open': request_open, 'request-close': request_close, + 'tu-vote-reminder': tu_vote_reminder, } db = mysql.connector.connect(host=aur_db_host, user=aur_db_user, diff --git a/scripts/tuvotereminder.py b/scripts/tuvotereminder.py new file mode 100755 index 0000000..0992623 --- /dev/null +++ b/scripts/tuvotereminder.py @@ -0,0 +1,32 @@ +#!/usr/bin/python3 + +import configparser +import mysql.connector +import os +import subprocess +import time + +config = configparser.RawConfigParser() +config.read(os.path.dirname(os.path.realpath(__file__)) + "/../conf/config") + +aur_db_host = config.get('database', 'host') +aur_db_name = config.get('database', 'name') +aur_db_user = config.get('database', 'user') +aur_db_pass = config.get('database', 'password') +aur_db_socket = config.get('database', 'socket') +notify_cmd = config.get('notifications', 'notify-cmd') + +db = mysql.connector.connect(host=aur_db_host, user=aur_db_user, + passwd=aur_db_pass, db=aur_db_name, + unix_socket=aur_db_socket, buffered=True) +cur = db.cursor() + +now = int(time.time()) +filter_from = now + 500 +filter_to = now + 172800 + +cur.execute("SELECT ID FROM TU_VoteInfo WHERE End >= %s AND End <= %s", + [filter_from, filter_to]) + +for vote_id in [row[0] for row in cur.fetchall()]: + subprocess.Popen((notify_cmd, 'tu-vote-reminder', str(vote_id))) -- 2.9.3

1 1

[aur-dev] Code complexity survey
by Vard Antinyan 27 Sep '16

27 Sep '16

Dear developers, We have undertaken a task to assess code complexity triggers and generate recommendations for developing understandable code. Our intension is to share the results with you, developers, so everyone can learn the triggers behind complex software. We need your help for rigorous results. My request to you is - if you get 5-10 min. time, would you please consider to answer the questions of this survey on code complexity? https://goo.gl/forms/h9WXZ8VSEw7BUyHg1 You are welcome to learn preliminary results through this link: https://www.facebook.com/SoftwareCodeQuality/photos/?tab=album&album_id=163… The results will be shared in a public webpage and everyone possible will be invited to learn and discuss them. Your knowledge and experience is vital for achieving substantial and generalizable results, and your effort is much appreciated! Sincerely Vard Antinyan PhD candidate in University of Gothenburg, Sweden Tel: 0046317725707

1 0

[aur-dev] [PATCH 00/17] Add tests for Python scripts
by Lukas Fleischer 26 Sep '16

26 Sep '16

Various changes in the way Python scripts and tests are organized. Shared Python modules are now placed in a site-packages directory. Tests for mkpkglists, tuvotereminder, pkgmaint and aurblup are added. Lukas Fleischer (17): Add a main() method to all Python scripts git-auth: Move entry point to a main() method git-serve: Pass user and privileges as parameters git-serve: Move entry point to a main() method git-update: Move entry point to a main() method Use setuptools to install Python modules Use config and db in scripts Make test suite paths top-level directory relative Reorganize tests Update README mkpkglists: Make output files configurable Add tests for mkpkglists Add tests for tuvotereminder scripts: Do not use UNIX_TIMESTAMP Add tests for pkgmaint aurblup: Drop support for multiple servers Add tests for aurblup README | 6 + aurweb/__init__.py | 0 {git-interface => aurweb}/config.py | 0 {git-interface => aurweb}/db.py | 16 +- conf/config.proto | 6 +- git-interface/__init__.py | 0 git-interface/git-auth.py | 58 ++-- git-interface/git-serve.py | 270 ++++++++++--------- git-interface/git-update.py | 300 +++++++++++---------- scripts/__init__.py | 0 scripts/aurblup.py | 105 ++++---- scripts/mkpkglists.py | 71 +++-- scripts/notify.py | 278 ++++++++++--------- scripts/pkgmaint.py | 31 +-- scripts/popupdate.py | 52 ++-- scripts/tuvotereminder.py | 40 ++- setup.py | 20 ++ {git-interface/test => test}/Makefile | 0 {git-interface/test => test}/setup.sh | 29 +- {git-interface/test => test}/sharness.sh | 0 .../test/t0001-auth.sh => test/t1100-git-auth.sh | 0 .../test/t0002-serve.sh => test/t1200-git-serve.sh | 0 .../t0003-update.sh => test/t1300-git-update.sh | 0 test/t2100-mkpkglists.sh | 47 ++++ test/t2200-tuvotereminder.sh | 26 ++ test/t2300-pkgmaint.sh | 26 ++ test/t2400-aurblup.sh | 53 ++++ 27 files changed, 818 insertions(+), 616 deletions(-) create mode 100644 aurweb/__init__.py rename {git-interface => aurweb}/config.py (100%) rename {git-interface => aurweb}/db.py (73%) create mode 100644 git-interface/__init__.py create mode 100644 scripts/__init__.py create mode 100644 setup.py rename {git-interface/test => test}/Makefile (100%) rename {git-interface/test => test}/setup.sh (89%) rename {git-interface/test => test}/sharness.sh (100%) rename git-interface/test/t0001-auth.sh => test/t1100-git-auth.sh (100%) rename git-interface/test/t0002-serve.sh => test/t1200-git-serve.sh (100%) rename git-interface/test/t0003-update.sh => test/t1300-git-update.sh (100%) create mode 100755 test/t2100-mkpkglists.sh create mode 100755 test/t2200-tuvotereminder.sh create mode 100755 test/t2300-pkgmaint.sh create mode 100755 test/t2400-aurblup.sh -- 2.10.0

1 17

[aur-dev] Can't update repo because source URL is too long
by Andreas 25 Sep '16

25 Sep '16

Today, I wanted to update caddy-full-bin (https://aur.archlinux.org/packages/caddy-full-bin/) but the push into AUR was rejected because the `.SRCINFO` check crashed: |remote: Traceback (most recent call last): remote: File "hooks/update", line 389, in <module> remote: save_metadata(metadata, conn, user) remote: File "hooks/update", line 135, in save_metadata remote: [pkgid, source_info['value'], source_info['arch']]) remote: File "/srv/http/vhosts/aur.archlinux.org/public/git-interface/db.py", line 43, in execute remote: cur.execute(query, params) remote: File "/usr/lib/python3.5/site-packages/mysql/connector/cursor.py", line 515, in execute remote: self._handle_result(self._connection.cmd_query(stmt)) remote: File "/usr/lib/python3.5/site-packages/mysql/connector/connection.py", line 488, in cmd_query remote: result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query)) remote: File "/usr/lib/python3.5/site-packages/mysql/connector/connection.py", line 395, in _handle_result remote: raise errors.get_exception(packet) remote: mysql.connector.errors.DataError: 1406 (22001): Data too long for column 'Source' at row 1 remote: error: hook declined to update refs/heads/master To git+ssh://aur.archlinux.org/caddy-full-bin.git| It seems that the column that stores source URLs in the database is to small: `|Data too long for column 'Source' at row 1|`. Here is an example source URL from the `.SRCINFO`: `|source_i686 = caddy.tar.gz::https://caddyserver.com/download/build?os=linux&features=awsl… |Caddys buildserver requires to pass all those parameters, therfore I can't reduce the length of the URL. There is an archlinux forum thread for discussing this issue: https://bbs.archlinux.org/viewtopic.php?id=217364 . | |Cheers, Andreas| -- Andreas Linz PGP: `F1E8 B1CC E0B7 FBF0 8E7E 33A2 A57E A370 7387 1146` klingt.net andreas-linz.de github.com/klingtnet

3 2

[aur-dev] [PATCH] Only show virtual dependency if version condition matches
by Johannes Löthberg 23 Sep '16

23 Sep '16

Signed-off-by: Johannes Löthberg <johannes(a)kyriasis.com> --- The vercmp logic is relatively complicated, so shelling out to the official vercmp binary is much simpler, and will be available on any system with pacman installed. conf/config.proto | 1 + web/lib/pkgfuncs.inc.php | 36 ++++++++++++++++++++++++++++++++++-- 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/conf/config.proto b/conf/config.proto index c56141c..37c455e 100644 --- a/conf/config.proto +++ b/conf/config.proto @@ -31,6 +31,7 @@ log_uri = https://aur.archlinux.org/cgit/aur.git/log/?h=%s snapshot_uri = /cgit/aur.git/snapshot/%s.tar.gz enable-maintenance = 1 maintenance-exceptions = 127.0.0.1 +vercmp-cmd = /usr/bin/vercmp [notifications] notify-cmd = /srv/http/aurweb/scripts/notify.py diff --git a/web/lib/pkgfuncs.inc.php b/web/lib/pkgfuncs.inc.php index 4b0fdba..a697e69 100644 --- a/web/lib/pkgfuncs.inc.php +++ b/web/lib/pkgfuncs.inc.php @@ -204,14 +204,14 @@ function pkg_groups($pkgid) { */ function pkg_providers($name) { $dbh = DB::connect(); - $q = "SELECT p.ID, p.Name FROM Packages p "; + $q = "SELECT p.ID, p.Name, p.Version FROM Packages p "; $q.= "LEFT JOIN PackageRelations pr ON pr.PackageID = p.ID "; $q.= "LEFT JOIN RelationTypes rt ON rt.ID = pr.RelTypeID "; $q.= "WHERE p.Name = " . $dbh->quote($name) . " "; $q.= "OR (rt.Name = 'provides' "; $q.= "AND pr.RelName = " . $dbh->quote($name) . ")"; $q.= "UNION "; - $q.= "SELECT 0, Name FROM OfficialProviders "; + $q.= "SELECT 0, Name, 0 FROM OfficialProviders "; $q.= "WHERE Provides = " . $dbh->quote($name); $result = $dbh->query($q); @@ -349,6 +349,33 @@ function pkg_provider_link($name, $official) { return $link; } +function cmp_provider_ver($version, $cond) { + if (!$cond) { + return True; + } + + $condition = preg_split("/^([<=>]{0,2})/", $cond, -1, PREG_SPLIT_DELIM_CAPTURE); + + $cmd = config_get('options', 'vercmp-cmd'); + $cmd .= ' ' . escapeshellarg($version) . ' ' . escapeshellarg($condition[2]); + $handle = popen($cmd, 'r'); + $result = fread($handle, 2); + pclose($handle); + + switch ($condition[1]) { + case ">=": + return $result >= 0; + case "<=": + return $result <= 0; + case ">": + return $result > 0; + case "<": + return $result < 0; + case "==": + return $result == 0; + } +} + /** * Get the HTML code to display a package dependency link * @@ -407,6 +434,11 @@ function pkg_depend_link($name, $type, $cond, $arch, $pkg_id) { if (count($providers) > 0) { $link .= '<span class="virtual-dep">('; foreach ($providers as $provider) { + $version = $provider[2]; + if (!cmp_provider_ver($version, $cond)) { + break; + } + $is_official = ($provider[0] == 0); $name = $provider[1]; $link .= pkg_provider_link($name, $is_official) . ', '; -- 2.10.0

1 0

[aur-dev] [PATCH 1/5] git-serve: Format usage text automatically
by Lukas Fleischer 18 Sep '16

18 Sep '16

Remove the formatting of the usage text and add code to columnize it automatically instead. Also, add more strict tests for the usage output. These new tests ensure that the usage header is printed, commands are indented properly and no overly long lines are produced. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-serve.py | 26 ++++++++++++++++++-------- git-interface/test/t0002-serve.sh | 10 +++++++++- 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/git-interface/git-serve.py b/git-interface/git-serve.py index 38048c9..0187de9 100755 --- a/git-interface/git-serve.py +++ b/git-interface/git-serve.py @@ -117,6 +117,14 @@ def warn(msg): sys.stderr.write("warning: {:s}\n".format(msg)) +def usage(cmds): + sys.stderr.write("Commands:\n") + colwidth = max([len(cmd) for cmd in cmds.keys()]) + 4 + for key in sorted(cmds): + sys.stderr.write(" " + key.ljust(colwidth) + cmds[key] + "\n") + exit(0) + + user = os.environ.get('AUR_USER') privileged = (os.environ.get('AUR_PRIVILEGED', '0') == '1') ssh_cmd = os.environ.get('SSH_ORIGINAL_COMMAND') @@ -187,13 +195,15 @@ elif action == 'restore': os.environ["AUR_PKGBASE"] = pkgbase os.execl(git_update_cmd, git_update_cmd, 'restore') elif action == 'help': - die("Commands:\n" + - " help Show this help message and exit.\n" + - " list-repos List all your repositories.\n" + - " restore <name> Restore a deleted package base.\n" + - " set-keywords <name> [...] Change package base keywords.\n" + - " setup-repo <name> Create a repository (deprecated).\n" + - " git-receive-pack Internal command used with Git.\n" + - " git-upload-pack Internal command used with Git.") + cmds = { + "help": "Show this help message and exit.", + "list-repos": "List all your repositories.", + "restore <name>": "Restore a deleted package base.", + "set-keywords <name> [...]": "Change package base keywords.", + "setup-repo <name>": "Create a repository (deprecated).", + "git-receive-pack": "Internal command used with Git.", + "git-upload-pack": "Internal command used with Git.", + } + usage(cmds) else: die_with_help("invalid command: {:s}".format(action)) diff --git a/git-interface/test/t0002-serve.sh b/git-interface/test/t0002-serve.sh index 52fdcd1..ce8340e 100755 --- a/git-interface/test/t0002-serve.sh +++ b/git-interface/test/t0002-serve.sh @@ -9,7 +9,15 @@ test_expect_success 'Test interactive shell.' ' ' test_expect_success 'Test help.' ' - SSH_ORIGINAL_COMMAND=help "$GIT_SERVE" 2>&1 | grep -q "^Commands:$" + SSH_ORIGINAL_COMMAND=help "$GIT_SERVE" 2>actual && + save_IFS=$IFS + IFS= + while read -r line; do + echo $line | grep -q "^Commands:$" && continue + echo $line | grep -q "^ [a-z]" || return 1 + [ ${#line} -le 80 ] || return 1 + done <actual + IFS=$save_IFS ' test_expect_success 'Test setup-repo and list-repos.' ' -- 2.9.3

1 4

[aur-dev] [PATCH] git-serve: Mark setup-repo as deprecated
by Lukas Fleischer 18 Sep '16

18 Sep '16

Since 0c1187c (git-serve: Deprecate setup-repo, 2016-07-24), it is no longer recommended to use setup-repo. Mark the command as deprecated in the usage/help text. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-serve.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/git-interface/git-serve.py b/git-interface/git-serve.py index 19c3ab2..38048c9 100755 --- a/git-interface/git-serve.py +++ b/git-interface/git-serve.py @@ -192,7 +192,7 @@ elif action == 'help': " list-repos List all your repositories.\n" + " restore <name> Restore a deleted package base.\n" + " set-keywords <name> [...] Change package base keywords.\n" + - " setup-repo <name> Create an empty repository.\n" + + " setup-repo <name> Create a repository (deprecated).\n" + " git-receive-pack Internal command used with Git.\n" + " git-upload-pack Internal command used with Git.") else: -- 2.9.3

1 0