Searching the AUR API by dependent packages?
Hi, I wanted to use the AUR API to list packages that depend on a specific package. But it seems that the API does not implement this kind of search. The use-case is the following: when I bump a package in [community], I would like to list all packages in the AUR that potentially need a rebuild. Would this be a welcome addition to the API? If it is, I may try to hack this myself (but I have no guarantee about available time, and I stopped writing PHP and raw SQL years ago) The API I have in mind is the following: /rpc/?type=search&by=depends&arg=ocaml → returns all packages that depend on ocaml /rpc/?type=search&by=makedepends&arg=ocaml → returns all packages that make-depend on ocaml /rpc/?type=search&by=optdepends&arg=ocaml → returns all packages that opt-depend on ocaml Thanks, Baptiste
From: Baptiste Jonglez <git@bitsofnetworks.org> It is now possible to search for packages that depend on a given package, for instance: /rpc/?v=5&type=search&by=depends&arg=ocaml It is also possible to match on "makedepends", "checkdepends" and "optdepends". Remaining tasks for a complete patch: - actually test it... - update documentation - see if the additional SQL JOINs increase database load - maybe add something like "by=anydepends" to match across all types of dependencies? Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org> --- web/lib/aurjson.class.php | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 9eeaafd..addb92e 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -17,7 +17,8 @@ class AurJSON { 'suggest-pkgbase', 'get-comment-form' ); private static $exposed_fields = array( - 'name', 'name-desc', 'maintainer' + 'name', 'name-desc', 'maintainer', + 'depends', 'makedepends', 'checkdepends', 'optdepends' ); private static $fields_v1 = array( 'Packages.ID', 'Packages.Name', @@ -278,6 +279,10 @@ class AurJSON { "ON PackageBases.ID = Packages.PackageBaseID " . "LEFT JOIN Users " . "ON PackageBases.MaintainerUID = Users.ID " . + "LEFT JOIN PackageDepends " . + "ON Packages.ID = PackageDepends.PackageID " . + "LEFT JOIN DependencyTypes " . + "ON PackageDepends.DepTypeID = DependencyTypes.ID " . "WHERE ${where_condition} " . "AND PackageBases.PackagerUID IS NOT NULL " . "LIMIT $max_results"; @@ -407,6 +412,15 @@ class AurJSON { $keyword_string = $this->dbh->quote($keyword_string); $where_condition = "Users.Username = $keyword_string "; } + } else if ($search_by === 'depends' || $search_by === 'makedepends' || $search_by === 'checkdepends' || $search_by === 'optdepends') { + if (empty($keyword_string)) { + return $this->json_error('Query arg is empty.'); + } else { + $keyword_string = $this->dbh->quote($keyword_string); + $search_string = $this->dbh->quote($search_by); + $where_condition = "PackageDepends.DepName = $keyword_string AND "; + $where_condition .= "DependencyTypes.Name = $search_string"; + } } return $this->process_query('search', $where_condition); -- 2.16.1
On Sun, 28 Jan 2018 at 01:20:55, Baptiste Jonglez wrote:
I like the overall idea and given the implementation is pretty simple and straightforward, I do not see anything speaking against it. The documentation definitely needs to be updated, preferably in the same patch. One small comment below...
This can be written in a more compact way, using in_array(). Everything else looks good to me! Thanks, Lukas
Hi Lukas, Thanks for the positive feedback! On 28-01-18, Lukas Fleischer wrote:
I like the overall idea and given the implementation is pretty simple and straightforward, I do not see anything speaking against it.
Actually, there was a mistake in the code I sent: the additional SQL JOIN causes the RPC to return duplicate results for the other types of search (name, name-desc and maintainer). Unfortunately, this makes the new code less straightforward. I have sent a new patch, can you please review the updated part?
The documentation definitely needs to be updated, preferably in the same patch.
Done in the new patch.
Updated in the new patch.
Everything else looks good to me!
Thanks! Baptiste
From: Baptiste Jonglez <git@bitsofnetworks.org> It is now possible to search for packages that depend on a given package, for instance: /rpc/?v=5&type=search&by=depends&arg=ocaml It is similarly possible to match on "makedepends", "checkdepends" and "optdepends". Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org> --- doc/rpc.txt | 8 +++++++- web/lib/aurjson.class.php | 33 +++++++++++++++++++++++++++++---- 2 files changed, 36 insertions(+), 5 deletions(-) diff --git a/doc/rpc.txt b/doc/rpc.txt index f353ff0..3148ebe 100644 --- a/doc/rpc.txt +++ b/doc/rpc.txt @@ -11,6 +11,10 @@ search argument and _field_ is one of the following values: * `name` (search by package name only) * `name-desc` (search by package name and description) * `maintainer` (search by package maintainer) +* `depends` (search for packages that depend on _keywords_) +* `makedepends` (search for packages that makedepend on _keywords_) +* `optdepends` (search for packages that optdepend on _keywords_) +* `checkdepends` (search for packages that checkdepend on _keywords_) The _by_ parameter can be skipped and defaults to `name-desc`. @@ -30,7 +34,9 @@ Examples `search`:: `/rpc/?v=5&type=search&arg=foobar` `search` by maintainer:: - `/rpc/?v=5&type=search&search_by=maintainer&arg=john` + `/rpc/?v=5&type=search&by=maintainer&arg=john` +`search` packages that have _boost_ as `makedepends`:: + `/rpc/?v=5&type=search&by=makedepends&arg=boost` `search` with callback:: `/rpc/?v=5&type=search&arg=foobar&callback=jsonp1192244621103` `info`:: diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 9eeaafd..30bdc89 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -17,7 +17,8 @@ class AurJSON { 'suggest-pkgbase', 'get-comment-form' ); private static $exposed_fields = array( - 'name', 'name-desc', 'maintainer' + 'name', 'name-desc', 'maintainer', + 'depends', 'makedepends', 'checkdepends', 'optdepends' ); private static $fields_v1 = array( 'Packages.ID', 'Packages.Name', @@ -243,16 +244,27 @@ class AurJSON { /* * Retrieve package information (used in info, multiinfo, search and - * msearch requests). + * depends requests). * * @param $type The request type. * @param $where_condition An SQL WHERE-condition to filter packages. + * @param $join_depends Whether to add a SQL JOIN on the PackageDepends table. + * It will produce duplicate packages unless $where_condition filters + * the result appropriately. * * @return mixed Returns an array of package matches. */ - private function process_query($type, $where_condition) { + private function process_query($type, $where_condition, $join_depends=false) { $max_results = config_get_int('options', 'max_rpc_results'); + $additional_joins = ""; + if ($join_depends) { + $additional_joins .= "LEFT JOIN PackageDepends " . + "ON Packages.ID = PackageDepends.PackageID " . + "LEFT JOIN DependencyTypes " . + "ON PackageDepends.DepTypeID = DependencyTypes.ID"; + } + if ($this->version == 1) { $fields = implode(',', self::$fields_v1); $query = "SELECT {$fields} " . @@ -264,6 +276,7 @@ class AurJSON { "ON PackageLicenses.PackageID = Packages.ID " . "LEFT JOIN Licenses " . "ON Licenses.ID = PackageLicenses.LicenseID " . + "${additional_joins} " . "WHERE ${where_condition} " . "AND PackageBases.PackagerUID IS NOT NULL " . "LIMIT $max_results"; @@ -278,6 +291,7 @@ class AurJSON { "ON PackageBases.ID = Packages.PackageBaseID " . "LEFT JOIN Users " . "ON PackageBases.MaintainerUID = Users.ID " . + "${additional_joins} " . "WHERE ${where_condition} " . "AND PackageBases.PackagerUID IS NOT NULL " . "LIMIT $max_results"; @@ -380,6 +394,7 @@ class AurJSON { * @return mixed Returns an array of package matches. */ private function search($http_data) { + $join_depends = false; $keyword_string = $http_data['arg']; if (isset($http_data['by'])) { @@ -407,9 +422,19 @@ class AurJSON { $keyword_string = $this->dbh->quote($keyword_string); $where_condition = "Users.Username = $keyword_string "; } + } else if (in_array($search_by, ['depends', 'makedepends', 'checkdepends', 'optdepends'])) { + if (empty($keyword_string)) { + return $this->json_error('Query arg is empty.'); + } else { + $keyword_string = $this->dbh->quote($keyword_string); + $search_string = $this->dbh->quote($search_by); + $where_condition = "PackageDepends.DepName = $keyword_string AND "; + $where_condition .= "DependencyTypes.Name = $search_string"; + $join_depends = true; + } } - return $this->process_query('search', $where_condition); + return $this->process_query('search', $where_condition, $join_depends); } /* -- 2.16.1
On Sun, 28 Jan 2018 at 22:27:34, Baptiste Jonglez wrote:
Great!
Just another idea: maybe we can put the new fields in a separate array $exposed_depfields and merge that array into $exposed_fields upon initialization. Then we would not need to hardcode them again in the in_array() check below.
This is a bit unfortunate, indeed. Can we, instead of doing this, use a subselect? Something like the following statement might work: $keyword_string IN (SELECT PackageDepends.DepName FROM PackageDepends LEFT JOIN DependencyTypes ON PackageDepends.DepTypeID = DependencyTypes.ID WHERE PackageDepends.PackageID = Packages.ID AND DependencyTypes.Name = $search_string) Note that this is fully untested, though. Best regards, Lukas
Thanks for the new review! I have sent a v2 PATCH, comments inline below. On 29-01-18, Lukas Fleischer wrote:
I have moved the new fields to a variable $exposed_depfields, it indeed makes things slightly easier to read. However, I think that merging the two arrays later in the code would be rather confusing for the reader (because at first glance, the dependency fields would seem to be absent from $exposed_fields).
Excellent idea, and it even worked on the first try! It simplifies the code a lot, and performance-wise it should be roughly similar to a JOIN, possibly a bit slower (from what I could read on various stack overflow posts). Baptiste
From: Baptiste Jonglez <git@bitsofnetworks.org> It is now possible to search for packages that depend on a given package, for instance: /rpc/?v=5&type=search&by=depends&arg=ocaml It is similarly possible to match on "makedepends", "checkdepends" and "optdepends". Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org> --- v2: - use a subquery instead of a conditional JOIN, to simplify the code - avoid hard-coding the new list of allowed fields deep inside the code doc/rpc.txt | 8 +++++++- web/lib/aurjson.class.php | 21 +++++++++++++++++++-- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/doc/rpc.txt b/doc/rpc.txt index f353ff0..3148ebe 100644 --- a/doc/rpc.txt +++ b/doc/rpc.txt @@ -11,6 +11,10 @@ search argument and _field_ is one of the following values: * `name` (search by package name only) * `name-desc` (search by package name and description) * `maintainer` (search by package maintainer) +* `depends` (search for packages that depend on _keywords_) +* `makedepends` (search for packages that makedepend on _keywords_) +* `optdepends` (search for packages that optdepend on _keywords_) +* `checkdepends` (search for packages that checkdepend on _keywords_) The _by_ parameter can be skipped and defaults to `name-desc`. @@ -30,7 +34,9 @@ Examples `search`:: `/rpc/?v=5&type=search&arg=foobar` `search` by maintainer:: - `/rpc/?v=5&type=search&search_by=maintainer&arg=john` + `/rpc/?v=5&type=search&by=maintainer&arg=john` +`search` packages that have _boost_ as `makedepends`:: + `/rpc/?v=5&type=search&by=makedepends&arg=boost` `search` with callback:: `/rpc/?v=5&type=search&arg=foobar&callback=jsonp1192244621103` `info`:: diff --git a/web/lib/aurjson.class.php b/web/lib/aurjson.class.php index 9eeaafd..8b760fb 100644 --- a/web/lib/aurjson.class.php +++ b/web/lib/aurjson.class.php @@ -17,7 +17,11 @@ class AurJSON { 'suggest-pkgbase', 'get-comment-form' ); private static $exposed_fields = array( - 'name', 'name-desc', 'maintainer' + 'name', 'name-desc', 'maintainer', + 'depends', 'makedepends', 'checkdepends', 'optdepends' + ); + private static $exposed_depfields = array( + 'depends', 'makedepends', 'checkdepends', 'optdepends' ); private static $fields_v1 = array( 'Packages.ID', 'Packages.Name', @@ -243,7 +247,7 @@ class AurJSON { /* * Retrieve package information (used in info, multiinfo, search and - * msearch requests). + * depends requests). * * @param $type The request type. * @param $where_condition An SQL WHERE-condition to filter packages. @@ -407,6 +411,19 @@ class AurJSON { $keyword_string = $this->dbh->quote($keyword_string); $where_condition = "Users.Username = $keyword_string "; } + } else if (in_array($search_by, self::$exposed_depfields)) { + if (empty($keyword_string)) { + return $this->json_error('Query arg is empty.'); + } else { + $keyword_string = $this->dbh->quote($keyword_string); + $search_by = $this->dbh->quote($search_by); + $subquery = "SELECT PackageDepends.DepName FROM PackageDepends "; + $subquery .= "LEFT JOIN DependencyTypes "; + $subquery .= "ON PackageDepends.DepTypeID = DependencyTypes.ID "; + $subquery .= "WHERE PackageDepends.PackageID = Packages.ID "; + $subquery .= "AND DependencyTypes.Name = $search_by"; + $where_condition = "$keyword_string IN ($subquery)"; + } } return $this->process_query('search', $where_condition); -- 2.16.1
On 28-01-18, Baptiste Jonglez wrote:
So actually, this was not so hard to implement: RFC patch is incoming. Review welcome! Baptiste
participants (2)
-
Baptiste Jonglez
-
Lukas Fleischer