- Aur-dev - lists.archlinux.org

Searching the AUR API by dependent packages?
by Baptiste Jonglez 23 Feb '18

23 Feb '18

Hi, I wanted to use the AUR API to list packages that depend on a specific package. But it seems that the API does not implement this kind of search. The use-case is the following: when I bump a package in [community], I would like to list all packages in the AUR that potentially need a rebuild. Would this be a welcome addition to the API? If it is, I may try to hack this myself (but I have no guarantee about available time, and I stopped writing PHP and raw SQL years ago) The API I have in mind is the following: /rpc/?type=search&by=depends&arg=ocaml → returns all packages that depend on ocaml /rpc/?type=search&by=makedepends&arg=ocaml → returns all packages that make-depend on ocaml /rpc/?type=search&by=optdepends&arg=ocaml → returns all packages that opt-depend on ocaml Thanks, Baptiste

2 9

[PATCH] Add rate limit support to API
by Florian Pritz 23 Feb '18

23 Feb '18

This allows us to prevent users from hammering the API every few seconds to check if any of their packages were updated. Real world users check as often as every 5 or 10 seconds. Signed-off-by: Florian Pritz <bluewind(a)xinu.at> --- Basic idea for a rate limiting solution. Currently the cleanup of old entries is done on each api request. That may be a bit excessive, but I didn't find a cronjob to put it and given the table is indexed, the query is probably fast enough. At least for a PoC this should be fine. Tell me what you think. conf/config.proto | 4 +++ upgrading/4.7.0.txt | 11 ++++++++ web/lib/aurjson.class.php | 71 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 86 insertions(+) create mode 100644 upgrading/4.7.0.txt diff --git a/conf/config.proto b/conf/config.proto index 1750929..934d369 100644 --- a/conf/config.proto +++ b/conf/config.proto @@ -36,6 +36,10 @@ enable-maintenance = 1 maintenance-exceptions = 127.0.0.1 render-comment-cmd = /usr/local/bin/aurweb-rendercomment +[ratelimit] +request_limit = 4000 +window_length = 86400 + [notifications] notify-cmd = /usr/local/bin/aurweb-notify sendmail = /usr/bin/sendmail diff --git a/upgrading/4.7.0.txt b/upgrading/4.7.0.txt new file mode 100644 index 0000000..1b51cd2 --- /dev/null +++ b/upgrading/4.7.0.txt @@ -0,0 +1,11 @@ +1. Add ApiRateLimit table: + +--- +CREATE TABLE `ApiRateLimit` ( + `ip` varchar(45) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + `requests` int(11) NOT NULL, + `window_start` bigint(20) NOT NULL, + PRIMARY KEY (`ip`), + KEY `window_start_idx` (`window_start`) +) ENGINE=InnoDB; +--- diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 9eeaafd..8a6dd7b 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -96,6 +96,11 @@ public function handle($http_data) { $this->dbh = DB::connect(); + if ($this->check_ratelimit($_SERVER['REMOTE_ADDR'])) { + header("HTTP/1.1 429 Too Many Requests"); + return $this->json_error('Rate limit reached'); + } + $type = str_replace('-', '_', $http_data['type']); if ($type == 'info' && $this->version >= 5) { $type = 'multiinfo'; @@ -130,6 +135,72 @@ public function handle($http_data) { } } + /* + * Check if an IP needs to be rate limited. + * + * @param $ip IP of the current request + * + * @return true if IP needs to be rate limited, false otherwise. + */ + private function check_ratelimit($ip) { + $limit = config_get("ratelimit", "request_limit"); + $window_length = config_get("ratelimit", "window_length"); + $this->update_ratelimit($ip); + $stmt = $this->dbh->prepare(" + SELECT requests,window_start FROM ApiRateLimit + WHERE ip = :ip"); + $stmt->bindParam(":ip", $ip); + $result = $stmt->execute(); + + if (!$result) { + return false; + } + + $row = $stmt->fetch(PDO::FETCH_ASSOC); + + if ($row['window_start'] < time() - $window_length) { + $stmt = $this->dbh->prepare(" + DELETE FROM ApiRateLimit + WHERE ip = :ip"); + $stmt->bindParam(":ip", $ip); + $stmt->execute(); + return false; + } + + if ($row['requests'] > $limit) { + return true; + } + return false; + } + + /* + * Update a rate limit for an IP by increasing it's requests value by one. + * + * @param $ip IP of the current request + * + * @return void + */ + private function update_ratelimit($ip) { + $window_length = config_get("ratelimit", "window_length"); + $time = time(); + $deletion_time = $time - $window_length; + $stmt = $this->dbh->prepare(" + INSERT INTO ApiRateLimit + (ip, requests, window_start) + VALUES (:ip, 0, :window_start) + ON DUPLICATE KEY UPDATE requests=requests+1"); + $stmt->bindParam(":ip", $ip); + $stmt->bindParam(":window_start", $time); + $stmt->execute(); + + // TODO: this should move into some cronjob + $stmt = $this->dbh->prepare(" + DELETE FROM ApiRateLimit + WHERE window_start < :time"); + $stmt->bindParam(":time", $deletion_time); + $stmt->execute(); + } + /* * Returns a JSON formatted error string. * -- 2.15.1

2 5

[PATCH] Remove disjunction in pkg_providers query
by Florian Pritz 23 Feb '18

23 Feb '18

For some reason, running the SELECT .. WHERE .. OR .. query takes e.g. 58ms on a randomly generated db for some dependency name. Splitting the OR into two dedicated queries and UNIONing the result takes only 0.42ms. On the Arch Linux installation, searching for the providers of e.g. mongodb takes >=110ms when not cached by the query cache. The new query takes <1ms even when not cached. Signed-off-by: Florian Pritz <bluewind(a)xinu.at> --- web/lib/pkgfuncs.inc.php | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/web/lib/pkgfuncs.inc.php b/web/lib/pkgfuncs.inc.php index d022ebe..1dd8481 100644 --- a/web/lib/pkgfuncs.inc.php +++ b/web/lib/pkgfuncs.inc.php @@ -212,10 +212,12 @@ function pkg_groups($pkgid) { function pkg_providers($name) { $dbh = DB::connect(); $q = "SELECT p.ID, p.Name FROM Packages p "; + $q.= "WHERE p.Name = " . $dbh->quote($name) . " "; + $q.= "UNION "; + $q = "SELECT p.ID, p.Name FROM Packages p "; $q.= "LEFT JOIN PackageRelations pr ON pr.PackageID = p.ID "; $q.= "LEFT JOIN RelationTypes rt ON rt.ID = pr.RelTypeID "; - $q.= "WHERE p.Name = " . $dbh->quote($name) . " "; - $q.= "OR (rt.Name = 'provides' "; + $q.= "WHERE (rt.Name = 'provides' "; $q.= "AND pr.RelName = " . $dbh->quote($name) . ")"; $q.= "UNION "; $q.= "SELECT 0, Name FROM OfficialProviders "; -- 2.16.2

3 2

[PATCH] Add missing instruction for testing aurweb with a sqlite database
by Baptiste Jonglez 28 Jan '18

28 Jan '18

From: Baptiste Jonglez <git(a)bitsofnetworks.org> Also clarify what to change in the configuration. Signed-off-by: Baptiste Jonglez <git(a)bitsofnetworks.org> --- TESTING | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/TESTING b/TESTING index d2c97d9..fca23d7 100644 --- a/TESTING +++ b/TESTING @@ -13,7 +13,9 @@ INSTALL. # pacman -S php php-sqlite sqlite -3) Prepare the testing database: +3) Uncomment "extension=pdo_sqlite" in /etc/php/php.ini + +4) Prepare the testing database: $ cd /path/to/aurweb/schema $ make @@ -21,12 +23,12 @@ INSTALL. $ sqlite3 ../aurweb.sqlite3 < aur-schema-sqlite.sql $ sqlite3 ../aurweb.sqlite3 < out.sql -4) Copy conf/config.proto to conf/config and adjust the configuration +5) Copy conf/config.proto to conf/config and adjust the configuration (pay attention to disable_http_login, enable_maintenance and aur_location). - Be sure to change backend to sqlite and name to the file location of your - created test database. + Be sure to set "backend" to "sqlite" and "name" to the file location of your + created test database. An absolute path is recommended. -5) Run the PHP built-in web server: +6) Run the PHP built-in web server: $ AUR_CONFIG='/path/to/aurweb/conf/config' php -S localhost:8080 -t /path/to/aurweb/web/html -- 2.16.1

2 2

[PATCH] Document required PHP extensions in php.ini
by Remy Marquis 26 Jan '18

26 Jan '18

To people unfamiliar with the code, it is not obvious that the pdo_* PHP extensions must be enabled. --- INSTALL | 2 ++ TESTING | 2 ++ 2 files changed, 4 insertions(+) diff --git a/INSTALL b/INSTALL index ba24a45..7fcf724 100644 --- a/INSTALL +++ b/INSTALL @@ -38,6 +38,8 @@ read the instructions below. } } + Ensure to enable the pdo_mysql extension in php.ini. + 3) Copy conf/config.proto to /etc/aurweb/config and adjust the configuration (pay attention to disable_http_login, enable_maintenance and aur_location). diff --git a/TESTING b/TESTING index d2c97d9..e45f35f 100644 --- a/TESTING +++ b/TESTING @@ -13,6 +13,8 @@ INSTALL. # pacman -S php php-sqlite sqlite + Ensure to enable the pdo_sqlite extension in php.ini. + 3) Prepare the testing database: $ cd /path/to/aurweb/schema -- 2.15.1

2 1

[PATCH] Move git privilege check to update.py rather than auth.py
by Johannes Löthberg 21 Jan '18

21 Jan '18

auth.py is run as an AutherizedKeysCommand which does not get the environment variables passed to it, so AUR_OVERWRITE always got hard-set to '0' by it. Signed-off-by: Johannes Löthberg <johannes(a)kyriasis.com> --- aurweb/git/auth.py | 1 - aurweb/git/update.py | 2 +- test/t1100-git-auth.sh | 17 ----------------- 3 files changed, 1 insertion(+), 19 deletions(-) diff --git a/aurweb/git/auth.py b/aurweb/git/auth.py index b7819a9..828ed4e 100755 --- a/aurweb/git/auth.py +++ b/aurweb/git/auth.py @@ -53,7 +53,6 @@ def main(): env_vars = { 'AUR_USER': user, 'AUR_PRIVILEGED': '1' if account_type > 1 else '0', - 'AUR_OVERWRITE' : os.environ.get('AUR_OVERWRITE', '0') if account_type > 1 else '0', } key = keytype + ' ' + keytext diff --git a/aurweb/git/update.py b/aurweb/git/update.py index f681ddb..da48eb3 100755 --- a/aurweb/git/update.py +++ b/aurweb/git/update.py @@ -238,7 +238,7 @@ def main(): user = os.environ.get("AUR_USER") pkgbase = os.environ.get("AUR_PKGBASE") privileged = (os.environ.get("AUR_PRIVILEGED", '0') == '1') - allow_overwrite = (os.environ.get("AUR_OVERWRITE", '0') == '1') + allow_overwrite = (os.environ.get("AUR_OVERWRITE", '0') == '1') and privileged warn_or_die = warn if privileged else die if len(sys.argv) == 2 and sys.argv[1] == "restore": diff --git a/test/t1100-git-auth.sh b/test/t1100-git-auth.sh index dd20bea..71d526f 100755 --- a/test/t1100-git-auth.sh +++ b/test/t1100-git-auth.sh @@ -25,21 +25,4 @@ test_expect_success 'Test authentication with a wrong key.' ' test_must_be_empty out ' -test_expect_success 'Test AUR_OVERWRITE passthrough.' ' - AUR_OVERWRITE=1 \ - "$GIT_AUTH" "$AUTH_KEYTYPE_TU" "$AUTH_KEYTEXT_TU" >out && - grep -q AUR_OVERWRITE=1 out -' - -test_expect_success 'Make sure that AUR_OVERWRITE is unset by default.' ' - "$GIT_AUTH" "$AUTH_KEYTYPE_TU" "$AUTH_KEYTEXT_TU" >out && - grep -q AUR_OVERWRITE=0 out -' - -test_expect_success 'Make sure regular users cannot set AUR_OVERWRITE.' ' - AUR_OVERWRITE=1 \ - "$GIT_AUTH" "$AUTH_KEYTYPE_USER" "$AUTH_KEYTEXT_USER" >out && - grep -q AUR_OVERWRITE=0 out -' - test_done -- 2.16.0

2 4

[aur-dev][PATCH] Fix regression that stopped package maintainers from pinning comments
by Eli Schwartz 23 Dec '17

23 Dec '17

In commit 8c98db0b82cc85a4498589e5d60299fefd93b421 support was added for package comaintainers to pin comments in addition to maintainers. Due to a typo, the SQL query was reset halfway through and only added the comaintainer IDs to the list of allowed users. Fixes FS#56783 Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> --- web/lib/pkgfuncs.inc.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/lib/pkgfuncs.inc.php b/web/lib/pkgfuncs.inc.php index dbcf63e..d022ebe 100644 --- a/web/lib/pkgfuncs.inc.php +++ b/web/lib/pkgfuncs.inc.php @@ -100,7 +100,7 @@ function can_pin_comment($comment_id=0) { $q.= "LEFT JOIN PackageComments AS pc ON pb.ID = pc.PackageBaseID "; $q.= "WHERE pc.ID = " . intval($comment_id) . " "; $q.= "UNION "; - $q = "SELECT pcm.UsersID FROM PackageComaintainers AS pcm "; + $q.= "SELECT pcm.UsersID FROM PackageComaintainers AS pcm "; $q.= "LEFT JOIN PackageComments AS pc "; $q.= "ON pcm.PackageBaseID = pc.PackageBaseID "; $q.= "WHERE pc.ID = " . intval($comment_id); -- 2.15.1

2 1

aur-dev mailing list configuration changes regarding subject prefix and Reply-To header
by Florian Pritz 04 Dec '17

04 Dec '17

Hi, The aur-dev mailing list will no longer change common headers in mails that are being distributed. This especially means that the subject will no longer contain the [aur-dev] prefix and the list will not rewrite the reply-to header. If you want to filter mails coming from this list, please use the List-Id header. If you wish to reply to a mail from the list, use the "Reply to List" function of your mail client if available. This changes ensure that DKIM signatures remain valid if a mail passes through the list. If the mails were changed and the sender domain publishes a DMARC policy an invalid signature would lead to the mail being rejected by providers checking DMARC. To prevent this, the list previously rewrote the sender, but this makes it difficult to import patches with git am. By Lukas' request the list behaviour is now changed. If you experience any problems with this change, please feel free to contact me. Florian

1 1

[aur-dev] [PATCH] Allow setting an empty home page
by Lukas Fleischer 03 Dec '17

03 Dec '17

Since commit 4efba18 (Only allow valid HTTP(s) URLs as home page, 2017-11-05), the home page field in the account settings must be a valid URL. However, this new check prevents from leaving the field empty. Keep the check in place but skip it if the home page field is left empty. Fixes FS#56550. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- web/lib/acctfuncs.inc.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/lib/acctfuncs.inc.php b/web/lib/acctfuncs.inc.php index b8d9dc5..df57375 100644 --- a/web/lib/acctfuncs.inc.php +++ b/web/lib/acctfuncs.inc.php @@ -162,7 +162,7 @@ function process_account_form($TYPE,$A,$U="",$T="",$S="",$E="",$H="",$P="",$C="" $error = __("The email address is invalid."); } - if (!$error && !valid_homepage($HP)) { + if (!$error && !empty($HP) && !valid_homepage($HP)) { $error = __("The home page is invalid, please specify the full HTTP(s) URL."); } -- 2.15.1

1 0

[aur-dev] aurweb 4.6.0 released
by Lukas Fleischer 03 Dec '17

03 Dec '17

Dear aurweb contributors and users, We are pleased to announce the release of aurweb 4.6.0! This release brings a lot of improvements to the comment functionality. You can now use basic Markdown syntax to format your comments. Commit hashes to the Git repository of the package and references to Flyspray tickets are converted to links automatically. Long comments are collapsed and can be expanded on demand. We also made some changes not immediately visible to the end user, including several security fixes, a safety mechanism to prevent Trusted Users from accidentally performing non-fast-forward pushes and a public API to retrieve a list of registered account names. For a comprehensive list of changes, please consult the Git log [2]. As usual, bugs should be reported to the aurweb bug tracker [3]. [1] https://aur.archlinux.org/ [2] https://projects.archlinux.org/aurweb.git/log/?id=v4.6.0 [3] https://bugs.archlinux.org/index.php?project=2

1 0