April 2016 - Aur-dev - lists.archlinux.org

Re: [aur-dev] Making the AUR package list more useful
by Dave Reisner 26 May '16

26 May '16

On Apr 29, 2016 02:50, "Lukas Fleischer" <lfleischer(a)archlinux.org> wrote: > > Recently, there were a couple of feature requests to make the AUR > package search more powerful. While I do not plan on adding more > patterns or regular expressions to the RPC interface itself, my idea is > that more tools should be using the package name list. However, there > seem to be two issues with that: What are your reasons for not wanting to extend the RPC interface? Splitting the API in this way is cumbersome for clients. We now are supposed to download and maintain data which is essentially an index (which ought to be internal to the server to allow strong consistency). You're also asking clients of a read-only interface to become stateful, which isn't something that really interests me. > 1. The list is outdated. Right now, it is updated every two hours. I > do not think there is a good reason for those long intervals. > Reducing it to, say, ten minutes should be totally fine. Or maybe > even trigger list generation whenever a package is created or deleted > (which is clearly a lot more work, though). Thoughts? Or, change the storage for the name list such that updates can be fast. Turns out, you already have such a thing, you'd just need an index on the Packages and PackageBases tables. > 2. Transferring the whole package name list is inefficient. Even if we > use gzip compression here, the whole list is several hundreds of > kilobytes large. We need to retransfer the full list, even if only a > single package is added. Maybe we can do better than pacman here. My > idea is to add zsync support to the lists such that only relevant > parts are downloaded (for those who do not know: zsync is like rsync > but it works via HTTP as well and does not require any special > software on the server side). I did not experiment with how much > bandwidth we can actually save using this yet. Maybe the block size > needs to be adjusted. Are there any opinions or other suggestions on > this topic? I'm not understanding why any of this is considered a good direction for the API. What are the reasons for wanting the whole list of package names on the client side? Are there use cases other than search? > Regards, > Lukas

3 8

[aur-dev] Making the AUR package list more useful
by Lukas Fleischer 29 Apr '16

29 Apr '16

Recently, there were a couple of feature requests to make the AUR package search more powerful. While I do not plan on adding more patterns or regular expressions to the RPC interface itself, my idea is that more tools should be using the package name list. However, there seem to be two issues with that: 1. The list is outdated. Right now, it is updated every two hours. I do not think there is a good reason for those long intervals. Reducing it to, say, ten minutes should be totally fine. Or maybe even trigger list generation whenever a package is created or deleted (which is clearly a lot more work, though). Thoughts? 2. Transferring the whole package name list is inefficient. Even if we use gzip compression here, the whole list is several hundreds of kilobytes large. We need to retransfer the full list, even if only a single package is added. Maybe we can do better than pacman here. My idea is to add zsync support to the lists such that only relevant parts are downloaded (for those who do not know: zsync is like rsync but it works via HTTP as well and does not require any special software on the server side). I did not experiment with how much bandwidth we can actually save using this yet. Maybe the block size needs to be adjusted. Are there any opinions or other suggestions on this topic? Regards, Lukas

3 4

[aur-dev] [PATCH] Add hard limit for the length of dependency lists
by Lukas Fleischer 28 Apr '16

28 Apr '16

Introduce a configuration option max_depends which can be used to specify a maximum number of (reverse) dependencies to display on the package details pages. Fixes FS#49059. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- conf/config.proto | 1 + web/lib/pkgfuncs.inc.php | 10 ++++++---- web/template/pkg_details.php | 5 +++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/conf/config.proto b/conf/config.proto index 560c705..64af774 100644 --- a/conf/config.proto +++ b/conf/config.proto @@ -21,6 +21,7 @@ aur_location = https://aur.archlinux.org git_clone_uri_anon = https://aur.archlinux.org/%s.git git_clone_uri_priv = ssh://aur@aur.archlinux.org/%s.git max_rpc_results = 5000 +max_depends = 1000 aur_request_ml = aur-requests(a)archlinux.org request_idle_time = 1209600 auto_orphan_age = 15552000 diff --git a/web/lib/pkgfuncs.inc.php b/web/lib/pkgfuncs.inc.php index eaea318..4b0fdba 100644 --- a/web/lib/pkgfuncs.inc.php +++ b/web/lib/pkgfuncs.inc.php @@ -230,10 +230,11 @@ function pkg_providers($name) { * Get package dependencies for a specific package * * @param int $pkgid The package to get dependencies for + * @param int $limit An upper bound for the number of packages to retrieve * * @return array All package dependencies for the package */ -function pkg_dependencies($pkgid) { +function pkg_dependencies($pkgid, $limit) { $deps = array(); $pkgid = intval($pkgid); if ($pkgid > 0) { @@ -243,7 +244,7 @@ function pkg_dependencies($pkgid) { $q.= "OR SUBSTRING(pd.DepName FROM 1 FOR POSITION(': ' IN pd.DepName) - 1) = p.Name "; $q.= "LEFT JOIN DependencyTypes dt ON dt.ID = pd.DepTypeID "; $q.= "WHERE pd.PackageID = ". $pkgid . " "; - $q.= "ORDER BY pd.DepName"; + $q.= "ORDER BY pd.DepName LIMIT " . intval($limit); $result = $dbh->query($q); if (!$result) { return array(); @@ -505,10 +506,11 @@ function pkg_source_link($url, $arch) { * * @param string $name The package name for the dependency search * @param array $provides A list of virtual provisions of the package + * @param int $limit An upper bound for the number of packages to retrieve * * @return array All packages that depend on the specified package name */ -function pkg_required($name="", $provides) { +function pkg_required($name="", $provides, $limit) { $deps = array(); if ($name != "") { $dbh = DB::connect(); @@ -523,7 +525,7 @@ function pkg_required($name="", $provides) { $q.= "LEFT JOIN DependencyTypes dt ON dt.ID = pd.DepTypeID "; $q.= "WHERE pd.DepName IN (" . $name_list . ") "; $q.= "OR SUBSTRING(pd.DepName FROM 1 FOR POSITION(': ' IN pd.DepName) - 1) IN (" . $name_list . ") "; - $q.= "ORDER BY p.Name"; + $q.= "ORDER BY p.Name LIMIT " . intval($limit); $result = $dbh->query($q); if (!$result) {return array();} while ($row = $result->fetch(PDO::FETCH_NUM)) { diff --git a/web/template/pkg_details.php b/web/template/pkg_details.php index 8b038b9..b9c66d4 100644 --- a/web/template/pkg_details.php +++ b/web/template/pkg_details.php @@ -5,6 +5,7 @@ $log_uri = sprintf(config_get('options', 'log_uri'), urlencode($row['BaseName']) $snapshot_uri = sprintf(config_get('options', 'snapshot_uri'), urlencode($row['BaseName'])); $git_clone_uri_anon = sprintf(config_get('options', 'git_clone_uri_anon'), htmlspecialchars($row['BaseName'])); $git_clone_uri_priv = sprintf(config_get('options', 'git_clone_uri_priv'), htmlspecialchars($row['BaseName'])); +$max_depends = config_get_int('options', 'max_depends'); $uid = uid_from_sid($SID); @@ -40,7 +41,7 @@ $out_of_date_time = ($row["OutOfDateTS"] == 0) ? $msg : gmdate("Y-m-d", intval($ $lics = pkg_licenses($row["ID"]); $grps = pkg_groups($row["ID"]); -$deps = pkg_dependencies($row["ID"]); +$deps = pkg_dependencies($row["ID"], $max_depends); usort($deps, function($x, $y) { if ($x[1] != $y[1]) { @@ -82,7 +83,7 @@ foreach ($rels as $rel) { } } -$requiredby = pkg_required($row["Name"], $rels_p); +$requiredby = pkg_required($row["Name"], $rels_p, $max_depends); # $sources[0] = 'src'; $sources = pkg_sources($row["ID"]); -- 2.8.0

2 3

[aur-dev] Frequent ipv6 timeouts
by Colin Woodbury 21 Apr '16

21 Apr '16

Something is amiss. I'm the author of Aura, and I've had users come to me in droves recently saying that Aura: 1. Can't connect to the AUR 2. Doesn't recognize legitimate packages as AUR packages 3. Can't install/upgrade, etc And so I dove into my [http libraries]( http://hackage.haskell.org/package/aur) and tested the hell out of everything. After some tweaks, everything seemed to work. Until it didn't. More manual testing has shown me that when resolving requests to the RPC via ipv6, most calls timeout/hang. I'm seeing this through my libraries, through firefox, and through curl. However, when forcing the request to resolve via ipv4, it always succeeds: ``` curl -4 "https://aur.archlinux.org/rpc?v=5&type=search&arg=foobar" ``` The following forum posts touch on the same problem: * [Ssl and ipv6](https://bbs.archlinux.org/viewtopic.php?id=210932) * [AUR being screwy](https://bbs.archlinux.org/viewtopic.php?id=206820) Can anyone else confirm this behaviour? For the record, I'm on the west coast of North America.

1 0