[aur-general] New AUR Metadata Archives
Hello AUR users, In addition to pre-existing archives, we've introduced two new archives that can be used instead of bulk queries against the RPC. Pre-existing archives: - packages.gz - Listing of all packages separated by line break. - pkgbase.gz - Listing of all package bases separated by line break. - users.gz - Listening of all users separated by line break. Metadata archives: - packages-meta-v1.json.gz - A complete `type=search` formatted JSON package archive. - packages-meta-ext-v1.json.gz - A complete `type=multiinfo` formatted JSON package archive. All archives support Last-Modified and ETag. Each archive is updated on a rough ~5 minute interval. For any bulk users of the RPC, we ask that you consider these archives as a solution to repeated searches or bulk multiinfo requests. All archives are available for download at https://aur.archlinux.org/archive-name.gz Using these archives will drastically help the AUR with the amount of traffic required for API clients. Particularly with clients who are able to query by themselves en masse. We thank you all for contributing to the world of AUR and helping those who can use your maintained software as a result. Regards, Kevin -- Kevin Morris Software Developer Identities: - kevr @ Libera
On 10/11/2021 22:31, Kevin Morris via aur-general wrote:
Hello AUR users,
In addition to pre-existing archives, we've introduced two new archives that can be used instead of bulk queries against the RPC.
Pre-existing archives:
- packages.gz - Listing of all packages separated by line break. - pkgbase.gz - Listing of all package bases separated by line break. - users.gz - Listening of all users separated by line break.
Metadata archives:
- packages-meta-v1.json.gz - A complete `type=search` formatted JSON package archive. - packages-meta-ext-v1.json.gz - A complete `type=multiinfo` formatted JSON package archive.
All archives support Last-Modified and ETag. Each archive is updated on a rough ~5 minute interval. For any bulk users of the RPC, we ask that you consider these archives as a solution to repeated searches or bulk multiinfo requests.
All archives are available for download at https://aur.archlinux.org/archive-name.gz
Using these archives will drastically help the AUR with the amount of traffic required for API clients. Particularly with clients who are able to query by themselves en masse.
We thank you all for contributing to the world of AUR and helping those who can use your maintained software as a result.
Regards, Kevin
I see the metadata is in an array instead of NDJSON like I suggested. However Seems as the opening and closing array are on the first/last line I guess they could be trimmed out. Could I be safe doing this without the formatting changing?
Indeed, did try to keep this as streamable as possible for ya, but still 100% JSON. We of course could modify it if another method is better, but, if you can ignore the array lines and a trailing ',' on each package object line (except for the last), we're fine with keeping it like this. On Thu, Nov 11, 2021 at 12:15:28AM +0000, Morgan Adamiec via aur-general wrote:
On 10/11/2021 22:31, Kevin Morris via aur-general wrote:
Hello AUR users,
In addition to pre-existing archives, we've introduced two new archives that can be used instead of bulk queries against the RPC.
Pre-existing archives:
- packages.gz - Listing of all packages separated by line break. - pkgbase.gz - Listing of all package bases separated by line break. - users.gz - Listening of all users separated by line break.
Metadata archives:
- packages-meta-v1.json.gz - A complete `type=search` formatted JSON package archive. - packages-meta-ext-v1.json.gz - A complete `type=multiinfo` formatted JSON package archive.
All archives support Last-Modified and ETag. Each archive is updated on a rough ~5 minute interval. For any bulk users of the RPC, we ask that you consider these archives as a solution to repeated searches or bulk multiinfo requests.
All archives are available for download at https://aur.archlinux.org/archive-name.gz
Using these archives will drastically help the AUR with the amount of traffic required for API clients. Particularly with clients who are able to query by themselves en masse.
We thank you all for contributing to the world of AUR and helping those who can use your maintained software as a result.
Regards, Kevin
I see the metadata is in an array instead of NDJSON like I suggested. However Seems as the opening and closing array are on the first/last line I guess they could be trimmed out. Could I be safe doing this without the formatting changing?
-- Kevin Morris Software Developer Identities: - kevr @ Libera
Apologies; I misspoke -- This is the first I'm hearing about NDJSON. However, we would like this archive to be consumable by both someone who wants to consume content in a stream or someone that just wants to download a concrete JSON file. On Thu, Nov 11, 2021 at 12:15:28AM +0000, Morgan Adamiec via aur-general wrote:
On 10/11/2021 22:31, Kevin Morris via aur-general wrote:
Hello AUR users,
In addition to pre-existing archives, we've introduced two new archives that can be used instead of bulk queries against the RPC.
Pre-existing archives:
- packages.gz - Listing of all packages separated by line break. - pkgbase.gz - Listing of all package bases separated by line break. - users.gz - Listening of all users separated by line break.
Metadata archives:
- packages-meta-v1.json.gz - A complete `type=search` formatted JSON package archive. - packages-meta-ext-v1.json.gz - A complete `type=multiinfo` formatted JSON package archive.
All archives support Last-Modified and ETag. Each archive is updated on a rough ~5 minute interval. For any bulk users of the RPC, we ask that you consider these archives as a solution to repeated searches or bulk multiinfo requests.
All archives are available for download at https://aur.archlinux.org/archive-name.gz
Using these archives will drastically help the AUR with the amount of traffic required for API clients. Particularly with clients who are able to query by themselves en masse.
We thank you all for contributing to the world of AUR and helping those who can use your maintained software as a result.
Regards, Kevin
I see the metadata is in an array instead of NDJSON like I suggested. However Seems as the opening and closing array are on the first/last line I guess they could be trimmed out. Could I be safe doing this without the formatting changing?
-- Kevin Morris Software Developer Identities: - kevr @ Libera
Kevin, great -- thanks very much! i find the metadata archives very valuable. cheers, Greg
participants (3)
-
Greg Minshall
-
Kevin Morris
-
Morgan Adamiec