[arch-dev-public] [RFC] archweb nvchecker integration

older
[arch-dev-public] build automation...

Jelle van der Waa

31 Jan 2022 31 Jan '22

8:25 p.m.

We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb. nvchecker is a program which can monitor versions upstreams, github, pypi, haskell, crates.io or custom provided for updates. Various packagers already use nvchecker in some form. [1] The idea is that every package in svn has a .NVCHECKER file (linux/trunk/.NVCHECKER) which contains the package specific settings i.e.: [go] source = “github” github = “golang/go” prefix = “go” use_max_tag = true exclude_regex = “.(release|weekly|rc|alpha|beta).” Archweb will have a systemd service which regularly goes through the pkgbases, check if the file exists, runs nvchecker and if required flags the package out of date automatically. For Github sources a token is required so we can do 5000 requests per hour, Arch Linux already has a Github account so this is no issue. Felix already has a script which converts a subset of packages to nvchecker. This could be extended to apply to more packages. [2] The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks. [1] https://nvchecker.readthedocs.io/en/latest/usage.html#check-crates-io [2] https://github.com/felixonmars/archlinux-futils/blob/master/arch-to-nvchecke... Greetings, Jelle van der Waa

Attachments:

OpenPGP_signature.sig (application/pgp-signature — 488 bytes)

Show replies by date

George Rawlinson

1 Feb 1 Feb

3:55 a.m.

On 22-01-31 21:25, Jelle van der Waa via arch-dev-public wrote:

...

We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb.

nvchecker is a program which can monitor versions upstreams, github, pypi, haskell, crates.io or custom provided for updates. Various packagers already use nvchecker in some form. [1]

The idea is that every package in svn has a .NVCHECKER file (linux/trunk/.NVCHECKER) which contains the package specific settings i.e.:

[go] source = “github” github = “golang/go” prefix = “go” use_max_tag = true exclude_regex = “.(release|weekly|rc|alpha|beta).”

Archweb will have a systemd service which regularly goes through the pkgbases, check if the file exists, runs nvchecker and if required flags the package out of date automatically.

For Github sources a token is required so we can do 5000 requests per hour, Arch Linux already has a Github account so this is no issue.

Felix already has a script which converts a subset of packages to nvchecker. This could be extended to apply to more packages. [2]

The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

[1] https://nvchecker.readthedocs.io/en/latest/usage.html#check-crates-io [2] https://github.com/felixonmars/archlinux-futils/blob/master/arch-to-nvchecke...

Greetings,

Jelle van der Waa

I have no objections to this, and happily support this endeavour. However, I currently track 8 packages via shell scripts (available via nvchecker's "cmd" source), all found here[0]. Some of them require specific dependencies to be installed, e.g. html-xml-utils for normalising/extracting HTML elements. Would we be able to list specific dependencies in the .nvchecker files? [0]: https://git.little.kiwi/grawlinson/arch-pkgs/src/branch/primary/.repo/pkgver -- George Rawlinson

Morten Linderud

7:21 a.m.

On Tue, Feb 01, 2022 at 03:55:44AM +0000, George Rawlinson via arch-dev-public wrote:

...

I have no objections to this, and happily support this endeavour.

However, I currently track 8 packages via shell scripts (available via nvchecker's "cmd" source), all found here[0].

Some of them require specific dependencies to be installed, e.g. html-xml-utils for normalising/extracting HTML elements. Would we be able to list specific dependencies in the .nvchecker files?

[0]: https://git.little.kiwi/grawlinson/arch-pkgs/src/branch/primary/.repo/pkgver

These require running arbitrary scripts on archweb so they should probably remain unsupported. Is there no better options? -- Morten Linderud PGP: 9C02FF419FECBE16

George Rawlinson

8:54 p.m.

On 22-02-01 08:21, Morten Linderud via arch-dev-public wrote:

...

These require running arbitrary scripts on archweb so they should probably remain unsupported. Is there no better options?

At this stage, the following [community] packages that I maintain require massaging of HTML sources: * html-xml-utils * oil * parallel * libmilter (bundled with sendmail source) * time I suppose if a nvchecker plugin existed that utilised bs4 (beautiful soup), that would work. But I assume that would still fit your definition of "arbitrary script". :p -- George Rawlinson

Jelle van der Waa

2 Feb 2 Feb

5:15 p.m.

On 01/02/2022 21:54, George Rawlinson via arch-dev-public wrote:

...

On 22-02-01 08:21, Morten Linderud via arch-dev-public wrote:

...
These require running arbitrary scripts on archweb so they should probably remain unsupported. Is there no better options?

At this stage, the following [community] packages that I maintain require massaging of HTML sources:

* html-xml-utils * oil * parallel * libmilter (bundled with sendmail source) * time

I suppose if a nvchecker plugin existed that utilised bs4 (beautiful soup), that would work. But I assume that would still fit your definition of "arbitrary script". :p

Good that you bring this up, I'm leaning towards only support the default plugins for now. I think that already is an huge improvement over our current state. We can make an improvement issue and look what we can do about it :) Thanks, Jelle van der Waa

Felix Yan

7:03 p.m.

On 2/1/22 22:54, George Rawlinson via arch-dev-public wrote:

...

On 22-02-01 08:21, Morten Linderud via arch-dev-public wrote: At this stage, the following [community] packages that I maintain require massaging of HTML sources:

* html-xml-utils * oil * parallel * libmilter (bundled with sendmail source) * time

I suppose if a nvchecker plugin existed that utilised bs4 (beautiful soup), that would work. But I assume that would still fit your definition of "arbitrary script". :p

There is a regex plugin and a htmlparser plugin for this. The htmlparser plugin accepts XPath, but if you want to process it further the regex plugin may just work better. Examples for your packages: [html-xml-utils] source = "regex" url = "https://www.w3.org/Tools/HTML-XML-utils/" regex = "html-xml-utils-(.*?).tar.gz" [oil] source = "htmlparser" url = "https://www.oilshell.org/release/latest/" xpath = "//h1/text()" prefix = "Oil " -- Regards, Felix Yan

George Rawlinson

3 Feb 3 Feb

4:48 p.m.

On 22-02-02 21:03, Felix Yan via arch-dev-public wrote:

...

There is a regex plugin and a htmlparser plugin for this.

The htmlparser plugin accepts XPath, but if you want to process it further the regex plugin may just work better.

Examples for your packages:

[html-xml-utils] source = "regex" url = "https://www.w3.org/Tools/HTML-XML-utils/" regex = "html-xml-utils-(.*?).tar.gz"

[oil] source = "htmlparser" url = "https://www.oilshell.org/release/latest/" xpath = "//h1/text()" prefix = "Oil "

I am guilty of not reading the documentation. Thank you! :) -- George Rawlinson

Evangelos Foutras

2 Feb 2 Feb

5:32 p.m.

On Mon, 31 Jan 2022 at 22:27, Jelle van der Waa via arch-dev-public wrote:

...

The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

I'm not fond of the hidden all-caps filename for editable sources (whereas it's fine for .BUILDINFO and friends). More importantly though, has integration with [1] been considered? The best way to implement automatic version checking in Arch is to use existing infrastructure if possible. 🐱 [1] https://release-monitoring.org/

Levente Polyak

5:41 p.m.

On 2/2/22 18:32, Evangelos Foutras via arch-dev-public wrote:

...

On Mon, 31 Jan 2022 at 22:27, Jelle van der Waa via arch-dev-public wrote:

...
The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

I'm not fond of the hidden all-caps filename for editable sources (whereas it's fine for .BUILDINFO and friends). More importantly though, has integration with [1] been considered? The best way to implement automatic version checking in Arch is to use existing infrastructure if possible. 🐱

[1] https://release-monitoring.org/ cases.

Hi, thanks for the info. We have considered release monitoring but it has some notable disadvantages being disconnected from what we really may package in several cases like forks. There is a README in https://gitlab.archlinux.org/archlinux/sandcrawler Overall the nvchecker kind of approach is the best we can aim for if we do not want to re-invent the wheel. cheers, Levente

Leonidas Spyropoulos

5:42 p.m.

On 02/02/2022 17:32, Evangelos Foutras via arch-dev-public wrote:

...

I'm not fond of the hidden all-caps filename for editable sources (whereas it's fine for .BUILDINFO and friends). More importantly though, has integration with [1] been considered? The best way to implement automatic version checking in Arch is to use existing infrastructure if possible. 🐱

[1] https://release-monitoring.org/

I agree with foutrelis, rather have in lowercase, and preferably not hidden. On the subject of support plugins and / or scripts would something like the below be OK? ``` [intellij-idea-ua-eap] source = "cmd" cmd = "curl -s --location --header \"Accept: application/rdf+xml\" https://www.jetbrains.com/updates/updates.xml | xmllint --xpath \"string(/products/product[@name='IntelliJ IDEA']/channel[@status='release' or @status!='eap']/build/@fullNumber)\" -" ``` Packages required: curl, libxml2 -- Leonidas Spyropoulos PGP: 59E43E106B247368

Allan McRae

11:42 p.m.

On 3/2/22 03:42, Leonidas Spyropoulos via arch-dev-public wrote:

...

On 02/02/2022 17:32, Evangelos Foutras via arch-dev-public wrote:

...
I'm not fond of the hidden all-caps filename for editable sources (whereas it's fine for .BUILDINFO and friends). More importantly though, has integration with [1] been considered? The best way to implement automatic version checking in Arch is to use existing infrastructure if possible. 🐱

[1] https://release-monitoring.org/

I agree with foutrelis, rather have in lowercase, and preferably not hidden.

My recollection* of dbscripts is that a hidden file will not be copied to the release directory. Allan *Someone may want to look at the code and not rely on a memory from years ago...

Anatol Pomozov

5:59 p.m.

Hi On Wed, Feb 2, 2022 at 9:32 AM Evangelos Foutras via arch-dev-public < arch-dev-public@lists.archlinux.org> wrote:

...

On Mon, 31 Jan 2022 at 22:27, Jelle van der Waa via arch-dev-public wrote:

...
The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

I'm not fond of the hidden all-caps filename for editable sources (whereas it's fine for .BUILDINFO and friends). More importantly though, has integration with [1] been considered? The best way to implement automatic version checking in Arch is to use existing infrastructure if possible. 🐱

[1] https://release-monitoring.org/

And here is one more tool to check if package version is of out-of-date https://github.com/anatol/pkgoutofdate It is a pretty simple tool that does not require any modification to Arch repo. It simply tries to guess what is the next possible version and then checks the upstream download site for it. The drawback is that it does not handle slotted packages and projects that do not use semver release numbering. Here is an example of output from this tool: ompiler-rt: new version found - 13.0.0 => 13.0.1 linux-docs: new version found - 5.8 => 5.9 assimp: new version found - 5.1.4 => 5.1.5 lablgtk2: new version found - 2.18.11 => 2.18.12 python-pyparsing: new version found - 2.4.7 => 3.0.0 beep: new version found - 1.4.9 => 1.4.10 bind: new version found - 9.16.25 => 9.17.0 sdl2_gfx: new version found - 1.0.4 => 1.1.0 python-resolvelib: new version found - 0.5.5 => 0.6.0 python-feedparser: new version found - 6.0.2 => 6.0.3 asymptote: new version found - 2.70 => 2.71 python-prometheus_client: new version found - 0.12.0 => 0.13.0 perl-net-http: new version found - 6.21 => 6.22 libcerf: new version found - 1.17 => 2.0 ocaml-topkg: new version found - 1.0.4 => 1.0.5 libstdc++5: new version found - 3.3.6 => 3.4.0 postgresql-old-upgrade: new version found - 12.8 => 12.9 python-wrapt: new version found - 1.12.1 => 1.13.0 gnucash-docs: new version found - 2.6.16 => 2.6.17 perl-libwww: new version found - 6.58 => 6.59 socat: new version found - 1.7.4.1 => 1.7.4.2 pgbouncer: new version found - 1.16.0 => 1.16.1 lua51: new version found - 5.1.5 => 5.2.0 ant: new version found - 1.10.11 => 1.10.12 nuget: new version found - 5.10.0 => 5.11.0 shaderc: new version found - 2021.3 => 2021.4 haskell-containers: new version found - 0.3.0.0 => 0.4.0.0 perl-business-ismn: new version found - 1.201 => 1.202 directx-headers: new version found - 1.4.9 => 1.4.10 nginx-prometheus-exporter: new version found - 0.9.0 => 0.10.0 python-django: new version found - 3.2.10 => 3.2.11 xapian-core: new version found - 1.4.18 => 1.4.19 dejagnu: new version found - 1.6.2 => 1.6.3 perl-data-compare: new version found - 1.25 => 1.26 mypaint-brushes1: new version found - 1.3.1 => 2.0.0 gcc: new version found - 11.1.0 => 11.2.0

Felix Yan

6:35 p.m.

On 2/2/22 19:59, Anatol Pomozov via arch-dev-public wrote:

...

And here is one more tool to check if package version is of out-of-date https://github.com/anatol/pkgoutofdate

It is a pretty simple tool that does not require any modification to Arch repo. It simply tries to guess what is the next possible version and then checks the upstream download site for it. The drawback is that it does not handle slotted packages and projects that do not use semver release numbering.

According to my past experience with it, there is an important additional drawback: it often detects pre-releases and broken (withdrawn) releases (as marked in a web page, github releases, or the corresponding repository) and the mechanism has no way to tell about this. I am also a regular user of this tool but unfortunately I don't think it's a good idea to use as the main tool. It's a nice addition to my nvchecker config to occasionally check the consistency among different upstream sources (like new version in PyPI but there is no corresponding git tag, or maybe the PyPI package has been moved to a different repository, etc). I think it would still be better to be explicit here. The maintainer should decide about which source to use. -- Regards, Felix Yan

Sven-Hendrik Haase

4 Feb 4 Feb

4:47 a.m.

On 31.01.22 21:25, Jelle van der Waa via arch-dev-public wrote:

...

We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb.

nvchecker is a program which can monitor versions upstreams, github, pypi, haskell, crates.io or custom provided for updates. Various packagers already use nvchecker in some form. [1]

The idea is that every package in svn has a .NVCHECKER file (linux/trunk/.NVCHECKER) which contains the package specific settings i.e.:

[go] source = “github” github = “golang/go” prefix = “go” use_max_tag = true exclude_regex = “.(release|weekly|rc|alpha|beta).”

Archweb will have a systemd service which regularly goes through the pkgbases, check if the file exists, runs nvchecker and if required flags the package out of date automatically.

For Github sources a token is required so we can do 5000 requests per hour, Arch Linux already has a Github account so this is no issue.

Felix already has a script which converts a subset of packages to nvchecker. This could be extended to apply to more packages. [2]

The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

[1] https://nvchecker.readthedocs.io/en/latest/usage.html#check-crates-io [2] https://github.com/felixonmars/archlinux-futils/blob/master/arch-to-nvchecke...

Greetings,

Jelle van der Waa

I got nothing to add to what the others have said but I just wanted to say that this is awesome and thank you for the effort. :) Sven

David Runge

6 Feb 6 Feb

1:25 p.m.

On 2022-01-31 21:25:41 (+0100), Jelle van der Waa via arch-dev-public wrote:

...

We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb.

One thing that I have been wondering about is the following: How do you plan to integrate this with existing packager workflows? When I look at my own: I periodically (at least daily) check for updates to upstreams using nvchecker in a multi-repository (i.e. multiple nvchecker config files) setup and am usually aware of changes quite fast. If this is integrated into the website, are people supposed to only rely on the website, or would there be a way to query the status of a packager's packages via some CLI tooling (as that is what I am relying on currently and which is easily accessible from multiple locations)?

...

The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

I'm fine with the file format and addition. Thanks for looking into this! It is much appreciated! :) Best, David -- https://sleepmap.de

Levente Polyak

5:25 p.m.

On 2/6/22 14:25, David Runge via arch-dev-public wrote:

...

On 2022-01-31 21:25:41 (+0100), Jelle van der Waa via arch-dev-public wrote:

...
We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb.

One thing that I have been wondering about is the following: How do you plan to integrate this with existing packager workflows? When I look at my own: I periodically (at least daily) check for updates to upstreams using nvchecker in a multi-repository (i.e. multiple nvchecker config files) setup and am usually aware of changes quite fast.

If this is integrated into the website, are people supposed to only rely on the website, or would there be a way to query the status of a packager's packages via some CLI tooling (as that is what I am relying on currently and which is easily accessible from multiple locations)?

We have a lot of info exposed in archweb API that some cli scripts can pull and display. This would certainly make a lot of sense to expose as well :) cheers, Levente

Frederik Schwan

4:20 p.m.

On Mon, Jan 31, 2022 at 09:25:41PM +0100, Jelle van der Waa via arch-dev-public wrote:

...

We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb.

nvchecker is a program which can monitor versions upstreams, github, pypi, haskell, crates.io or custom provided for updates. Various packagers already use nvchecker in some form. [1]

The idea is that every package in svn has a .NVCHECKER file (linux/trunk/.NVCHECKER) which contains the package specific settings i.e.:

[go] source = “github” github = “golang/go” prefix = “go” use_max_tag = true exclude_regex = “.(release|weekly|rc|alpha|beta).”

Archweb will have a systemd service which regularly goes through the pkgbases, check if the file exists, runs nvchecker and if required flags the package out of date automatically.

For Github sources a token is required so we can do 5000 requests per hour, Arch Linux already has a Github account so this is no issue.

Felix already has a script which converts a subset of packages to nvchecker. This could be extended to apply to more packages. [2]

The question is if anyone object to checking these small .NVCHECKER files into our svn repository. If there are no real objections, I'll start implementing this functionality into archweb in two weeks.

[1] https://nvchecker.readthedocs.io/en/latest/usage.html#check-crates-io [2] https://github.com/felixonmars/archlinux-futils/blob/master/arch-to-nvchecke...

Greetings,

Jelle van der Waa

Using nvchecker for a couple of years now not only for Arch packages, but also for a lot of IoT devices, hobby equipment and more. Pretty happy with nvchecker-notify :) An integration into Archweb would be awesome. Do we remove the ability to mark packages out-of-date manually then? It seems to me that a broken nvchecker config in a certain repo is more a bug than an ood package after such an addition. Best regards, Frederik

Jelle van der Waa

7 Feb 7 Feb

7:01 p.m.

On 06/02/2022 17:20, Frederik Schwan via arch-dev-public wrote:

...

On Mon, Jan 31, 2022 at 09:25:41PM +0100, Jelle van der Waa via arch-dev-public wrote:

...
We’ve wanted automatic flagging packages out of date for a while and currently every packager has to come up with it’s own solution. Let’s solve this centralized by integrating nvchecker into archweb.

Using nvchecker for a couple of years now not only for Arch packages, but also for a lot of IoT devices, hobby equipment and more. Pretty happy with nvchecker-notify :)

An integration into Archweb would be awesome. Do we remove the ability to mark packages out-of-date manually then? It seems to me that a broken nvchecker config in a certain repo is more a bug than an ood package after such an addition.

As nvchecker would be opt-in, I think it makes sense to keep the manual flagging out of date for now. Depending on how good nvchecker works in general we could consider removing that option. Greetings, Jelle van der Waa

808

Age (days ago)

815

Last active (days ago)

List overview

Download

17 comments

12 participants

participants (12)

Allan McRae
Anatol Pomozov
David Runge
Evangelos Foutras
Felix Yan
Frederik Schwan
George Rawlinson
Jelle van der Waa
Leonidas Spyropoulos
Levente Polyak
Morten Linderud
Sven-Hendrik Haase

[arch-dev-public] [RFC] archweb nvchecker integration

tags

participants (12)