Hello AUR users,
First of all, I want to apologize: this was deployed about 6 hours
ago without a prior announcement. This was completely my fault and
done in a bit of haste. There are a few bugs seen so far which have
been worked out and deployed, but there are some final patches waiting
to be deployed on the morning of 2022-02-05, 6-8 hours from now.
There are likely to be some other issues we have not yet encountered.
We have introduced an error reporting system to let us know when these
occur; if users hit an internal server error, a traceback will be
logged out and a notification sent to a devops postmaster. They
can then locate the traceback by a particular ID. So, as users run
into serious issues, we'll get information about them automatically.
That being said, I desperately ask that users who encounter bugs
from here-on out which are not the result of an internal server error
report them to us at https://gitlab.archlinux.org/archlinux/aurweb/-/issues
using the Bug template. This kind of bug reporting will really help us.
Now, on to what's changed for users:
With the v6.0.0 release, aurweb now runs on Python using the
Starlette/FastAPI framework. While a complete list of changes
from v6.0.0 up to the current patch (v6.0.6) can be viewed at
https://gitlab.archlinux.org/archlinux/aurweb/-/tags, I'd like
to show a few notable changes at the bottom of this message.
Regarding there have been some modifications to the RPC API endpoints
which have broken some clients in AUR helpers. Some AUR helpers were
using the `/rpc.php/` endpoint to query the RPC, but this was removed
with the advent of v6.0.0.
There are two issues here: our Python code redirects trailing slashes
to their unslashed counterparts, and we did not support the `/rpc.php`
endpoint at all: we supported the `/rpc` endpoint.
These have been addressed in v6.0.6 with the following changes:
- https://gitlab.archlinux.org/kevr/aurweb/-/commit/0c1bd982ea0958722366bc5a1…
- https://gitlab.archlinux.org/kevr/aurweb/-/commit/101de8e7b18916dd84ae52d39…
A deprecation notice will be posted regarding `/rpc.php`. Clients
will have one month to migrate over to the `/rpc` endpoint, and
we'll continue to support non-redirected trailing slash queries.
Again, sorry for the inconvenience in that regard, and a few others.
Changes
=======
Release v6.0.0 - Python
This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.
Preface
-------
This v6.0.0 release makes the long-awaited Python port official.
Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.
Legend
------
There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:
- _self_
- Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
- Refers to a POST action which can be triggered via the relevent package
page at `/{pkgbase,packages}/{name}`.
Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:
- [A] Some feature that does something
- [A] The same feature where another thing has changed
Infrastructure
--------------
- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
missing both out of the box and integrally which force us to account
for the different database types. We now only support mysql, and should
be able to support postgresql without much effort in the future.
Note: Users wishing to easily spin up a database quickly can use
`docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
call these scripts when aurweb is installed into a poetry virtualenv.
HTML
----
- Pagers have all been modified. They still serve the same purpose, but
they have slightly different display.
- Some markup and methods around the website has been changed for
post requests, and some forms have been completely reworked.
Package Requests
----------------
- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
- Deletion
- Through `/packages` bulk delete action
- Through `/pkgbase/{name}/delete`
- Merge
- Through `/pkgbase/{name}/merge`
- Orphan
- Through `/packages` bulk disown action
- Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
if no pre-existing request exists. This was done to increase tracking of
package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
requests pertaining to the action performed. This comes with the downside
of multiple notifications sent out about a closure if more than one
request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere
Package Bulk Actions: /packages
-------------------------------
- The `Merge into` field has been removed. Merges now require being
performed via the `/pkgbase/{name}/merge` action.
Package View
------------
- Some cached metadata is no longer cached (pkginfo). Previously,
this was defaulted to a one day cache for some package information.
If we need to bring this back, we can.
TU Proposals
------------
- A valid username is now required for any addition or removal of a TU.
RPC
---
- `type=get-comment-form` has been removed and is now located at
`/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
should not heavily affect users, as a 3,000 record query is returned
in roughly 0.20ms from a local standpoint. We will be working on this
in aim to push it over PHP.
Archives
--------
- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
- Enable this by passing `--extended` to `aurweb-mkpkglists`.
Performance Changes
-------------------
As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:
- `/` loads much quicker as it is now persistently cached forcibly
for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
various pieces of package info for `cache_pkginfo_ttl`, which is
defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
We now query the database for requests related to the action based on
the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
- We are not satisfied with this. We'll be working on pushing this
over the edge along with the rest of the DB-intensive routes.
However, the speed degredation is quite negligible for users'
experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
on my local 4-core 8-thread system.
Upcoming Work
-------------
This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.
- Update request and tu vote pagers
[Missing] - SSO support for accounts.archlinux.org
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
--
Kevin Morris
Software & Linux Enthusiast
Hello RPC users,
There are a few problematic changes that have occurred with the aurweb
v6.0.0 release regarding the RPC that I'd like to address here.
With the v6.0.0 release, aurweb no longer uses PHP. Previously, the
Wiki has shown `/rpc.php` being used to query the RPC, which did work
and was completely valid. This was an oversight by me during the port.
The wiki has since been updated to reflect the current RPC API, which
hosts its documentation at https://aur.archlinux.org/rpc by providing
no arguments.
In addition, trailing slashes are now rewritten to their unslashed
counterparts.
As issues were discovered, I've patched a few things into the new
implementation: for RPC and only RPC, we will support non-redirected
trailing slashes. This removes the redirect issue from the equation
from clients that have encountered issues like yay and yaourt.
Issues:
- trailing slash redirects, a lot of clients don't follow redirects
- /rpc.php no longer works
Fixes:
- fix(rpc): remove trailing slash redirection
- https://gitlab.archlinux.org/kevr/aurweb/-/commit/0c1bd982ea0958722366bc5a1…
- temporarily support /rpc.php/?
- https://gitlab.archlinux.org/kevr/aurweb/-/commit/101de8e7b18916dd84ae52d39…
These changes will be deployed out to https://aur.archlinux.org within
several (6-8) hours from now. A follow-up reply will be given regarding
this.
Regarding `/rpc.php`, we will continue to support this for one month.
At that time (Fri 4th, Mar 2022), we will remove support for that
specific endpoint completely. This allows users to use their helpers
to upgrade themselves.
I'm very sorry for the inconvenience, and thanks to a ton of AUR
helpers for patching this quickly.
Regards,
Kevin
--
Kevin Morris
Software & Linux Enthusiast