Hello,
I rewrote stonecrest's python script which generated the Integrity Checks : http://projects.archlinux.org/?p=dbscripts.git;a=blob_plain;f=cron-jobs/chec... last results : http://www.archlinux.org/pipermail/arch-dev-public/2008-August/007655.html
Main changes include : * better and safer parsing of PKGBUILDs (it now relies on a separate bash script which source PKGBUILDs and generate python code, rather than trying to parse bash directly in python) * much better performance (thanks to a small python C extension which allows to call alpm_vercmp natively rather than having to fork the vercmp binary thousands of times) * more accurate dependency and provision handling (having worked on this part of the pacman code for a while probably helped me a bit here)
time ./check_archlinux.sh
=================== = Integrity Check = ===================
Performing integrity checks... ==> checking mismatches ==> checking archs ==> checking dependencies ==> checking makedepends
Missing Dependencies ---------------------- archboot --> 'bcm43xx-fwcutter>=006-2'
Missing Makedepends --------------------- wpa_supplicant --> 'kernel26<2.6.25' xosd --> 'xmms' flac --> 'xmms'
Summary --------- Invalid PKGBUILDs: 0 Mismatching PKGBUILD names: 0 Invalid archs: 0 Missing (make)dependencies: 4
./check_archlinux.sh 6,29s user 0,65s system 96% cpu 7,210 total
I had to apply this patch to actually execute the script because of my locale:
<patch> --- check_archlinux.sh.orig 2008-08-17 15:25:04.000000000 -0500 +++ check_archlinux.sh 2008-08-26 10:34:39.000000000 -0500 @@ -62,6 +62,7 @@ export CARCH
cat << EOF +# -*- coding: UTF-8 -*- class PacmanPackage: def __init__(self): self.name,self.version,self.path = "","","" </patch>
On Tue, Aug 26, 2008 at 2:21 AM, Xavier shiningxc@gmail.com wrote:
Hello,
I rewrote stonecrest's python script which generated the Integrity Checks : http://projects.archlinux.org/?p=dbscripts.git;a=blob_plain;f=cron-jobs/chec... last results : http://www.archlinux.org/pipermail/arch-dev-public/2008-August/007655.html
Main changes include :
- better and safer parsing of PKGBUILDs
(it now relies on a separate bash script which source PKGBUILDs and generate python code, rather than trying to parse bash directly in python)
- much better performance
(thanks to a small python C extension which allows to call alpm_vercmp natively rather than having to fork the vercmp binary thousands of times)
- more accurate dependency and provision handling
(having worked on this part of the pacman code for a while probably helped me a bit here)
time ./check_archlinux.sh
=================== = Integrity Check = ===================
Performing integrity checks... ==> checking mismatches ==> checking archs ==> checking dependencies ==> checking makedepends
Missing Dependencies
archboot --> 'bcm43xx-fwcutter>=006-2'
Missing Makedepends
wpa_supplicant --> 'kernel26<2.6.25' xosd --> 'xmms' flac --> 'xmms'
Summary
Invalid PKGBUILDs: 0 Mismatching PKGBUILD names: 0 Invalid archs: 0 Missing (make)dependencies: 4
./check_archlinux.sh 6,29s user 0,65s system 96% cpu 7,210 total
On Tue, Aug 26, 2008 at 5:39 PM, Gustavo A. Gómez Farhat gustavo.gomez.farhat@gmail.com wrote:
I had to apply this patch to actually execute the script because of my locale:
<patch> --- check_archlinux.sh.orig 2008-08-17 15:25:04.000000000 -0500 +++ check_archlinux.sh 2008-08-26 10:34:39.000000000 -0500 @@ -62,6 +62,7 @@ export CARCH
cat << EOF+# -*- coding: UTF-8 -*- class PacmanPackage: def __init__(self): self.name,self.version,self.path = "","",""
</patch>
Thanks for the feedback! I find this curious though : there is not any utf8 character in that generated packages.py file, is there? And what is your locale so that I can try to reproduce it?
My locale is es_CO.utf8... but now I am confused... I have rebuilt the script but now it works without the patch =)
On Tue, Aug 26, 2008 at 10:47 AM, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 5:39 PM, Gustavo A. Gómez Farhat gustavo.gomez.farhat@gmail.com wrote:
I had to apply this patch to actually execute the script because of my locale:
<patch> --- check_archlinux.sh.orig 2008-08-17 15:25:04.000000000 -0500 +++ check_archlinux.sh 2008-08-26 10:34:39.000000000 -0500 @@ -62,6 +62,7 @@ export CARCH
cat << EOF+# -*- coding: UTF-8 -*- class PacmanPackage: def __init__(self): self.name,self.version,self.path = "","",""
</patch>
Thanks for the feedback! I find this curious though : there is not any utf8 character in that generated packages.py file, is there? And what is your locale so that I can try to reproduce it?
Is there no maintaince on udev anymore?
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
Could you be even more obscure?
Are you simply referring to the fact that udev 119 is outdated? Did you at least try the latest version on your system and can confirm it works? And what are the advantages of the latest release? And do you have problems with the current packaged version? If so, which ones?
I hope you can answer all these questions, which should have been present in your original mail ...
On Tue, 26 Aug 2008 13:27:32 -0700, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
Could you be even more obscure?
Are you simply referring to the fact that udev 119 is outdated? Did you at least try the latest version on your system and can confirm it works? And what are the advantages of the latest release? And do you have problems with the current packaged version? If so, which ones?
I hope you can answer all these questions, which should have been present in your original mail ...
I was just wondering, it is outdated and I didn't see any udev release updates anymore. That's all.
There is no problem with the latest version, it works. Does Archlinux not try to be always up to date, soon us possible?
It's maintained ofcourse, although I do not believe any new release of udev has been present the last months. I see other distros (Ubulubuntebian) constantly updating it, but that seems to be distro-specific/internal changes/patches mostly.
Hey, why change something that works?
-- bjorn hamra (Disclaimer: I did little to no research prior to posting this. HA-HA.)
-----Original Message----- From: arch-general-bounces@archlinux.org [mailto:arch-general-bounces@archlinux.org] On Behalf Of Amanai Sent: 26. august 2008 22:35 To: General Discusson about Arch Linux Subject: Re: [arch-general] UDEV
On Tue, 26 Aug 2008 13:27:32 -0700, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
Could you be even more obscure?
Are you simply referring to the fact that udev 119 is outdated? Did you at least try the latest version on your system and
can confirm
it works? And what are the advantages of the latest release? And do you have problems with the current packaged version? If so, which ones?
I hope you can answer all these questions, which should have been present in your original mail ...
I was just wondering, it is outdated and I didn't see any udev release updates anymore. That's all.
There is no problem with the latest version, it works. Does Archlinux not try to be always up to date, soon us possible?
On Tue, Aug 26, 2008 at 10:35 PM, Amanai amanai@freenet.de wrote:
I was just wondering, it is outdated and I didn't see any udev release updates anymore. That's all.
There is no problem with the latest version, it works.
Cool, so what is so exciting in the latest version?
2008/8/26 Xavier shiningxc@gmail.com:
On Tue, Aug 26, 2008 at 10:35 PM, Amanai amanai@freenet.de wrote:
I was just wondering, it is outdated and I didn't see any udev release updates anymore. That's all.
There is no problem with the latest version, it works.
Cool, so what is so exciting in the latest version?
I wouldn't say there are *exciting* changes (these are rare for this type of software) but here are the latest changes v125 - http://lwn.net/Articles/291028/ v126 - http://lwn.net/Articles/292596/
I think udev wasn't updated mostly because Tobias who used to maintain it took vacation from Arch to enjoy the life. So not having new versions of udev is temporary.
On Tue, Aug 26, 2008 at 10:49 PM, Roman Kyrylych roman.kyrylych@gmail.com wrote:
I wouldn't say there are *exciting* changes (these are rare for this type of software) but here are the latest changes v125 - http://lwn.net/Articles/291028/ v126 - http://lwn.net/Articles/292596/
I think udev wasn't updated mostly because Tobias who used to maintain it took vacation from Arch to enjoy the life. So not having new versions of udev is temporary.
Thanks for the link. The arch version is actually 119, so the previous changes could be of interest too : http://git.kernel.org/?p=linux/hotplug/udev.git;a=blob;hb=HEAD;f=ChangeLog
There was a recent news story about udev and how they want people to just use the vanilla rules rather than go off naming all their devices in their own way.
http://www.computerworld.com.au/index.php/id;605162254;fp;4194304;fpid;1
C
On Tue, Aug 26, 2008 at 3:35 PM, Amanai amanai@freenet.de wrote:
On Tue, 26 Aug 2008 13:27:32 -0700, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
Could you be even more obscure?
Are you simply referring to the fact that udev 119 is outdated? Did you at least try the latest version on your system and can confirm it works? And what are the advantages of the latest release? And do you have problems with the current packaged version? If so, which ones?
I hope you can answer all these questions, which should have been present in your original mail ...
I was just wondering, it is outdated and I didn't see any udev release updates anymore. That's all.
There is no problem with the latest version, it works. Does Archlinux not try to be always up to date, soon us possible?
This is my fault. I grabbed maintainership of udev when Tobias took a vacation, and real life has hit me like a sack of bricks, so I have little time these days to do much.
This Wed afternoon I plan on running through all my outdated packages and updating them. I will make udev a priority.
Cheers, Aaron
Aaron Griffin schrieb:
On Tue, Aug 26, 2008 at 3:35 PM, Amanai amanai@freenet.de wrote:
On Tue, 26 Aug 2008 13:27:32 -0700, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
This is my fault. I grabbed maintainership of udev when Tobias took a vacation, and real life has hit me like a sack of bricks, so I have little time these days to do much.
This Wed afternoon I plan on running through all my outdated packages and updating them. I will make udev a priority.
Cheers, Aaron
Udev is a ton of work. Right now, we use a mix of udev upstream rules and our own rules, and we should probably switch to upstream rules as much as possible. But that means lots of testing and stuff and lots of reading rules and see if they are right for Arch. And of course, updating klibc-udev to the same state as udev. I wanted to do it once, then didn't have the time.
On Wed, Aug 27, 2008 at 3:30 PM, Thomas Bächler thomas@archlinux.org wrote:
Aaron Griffin schrieb:
On Tue, Aug 26, 2008 at 3:35 PM, Amanai amanai@freenet.de wrote:
On Tue, 26 Aug 2008 13:27:32 -0700, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
This is my fault. I grabbed maintainership of udev when Tobias took a vacation, and real life has hit me like a sack of bricks, so I have little time these days to do much.
This Wed afternoon I plan on running through all my outdated packages and updating them. I will make udev a priority.
Cheers, Aaron
Udev is a ton of work. Right now, we use a mix of udev upstream rules and our own rules, and we should probably switch to upstream rules as much as possible. But that means lots of testing and stuff and lots of reading rules and see if they are right for Arch. And of course, updating klibc-udev to the same state as udev. I wanted to do it once, then didn't have the time.
Basically, that's why the faster the better. I could be a tester if there is one to test.
On Wed, Aug 27, 2008 at 2:30 AM, Thomas Bächler thomas@archlinux.org wrote:
Aaron Griffin schrieb:
On Tue, Aug 26, 2008 at 3:35 PM, Amanai amanai@freenet.de wrote:
On Tue, 26 Aug 2008 13:27:32 -0700, Xavier shiningxc@gmail.com wrote:
On Tue, Aug 26, 2008 at 6:46 PM, Amanai amanai@freenet.de wrote:
Is there no maintaince on udev anymore?
This is my fault. I grabbed maintainership of udev when Tobias took a vacation, and real life has hit me like a sack of bricks, so I have little time these days to do much.
This Wed afternoon I plan on running through all my outdated packages and updating them. I will make udev a priority.
Cheers, Aaron
Udev is a ton of work. Right now, we use a mix of udev upstream rules and our own rules, and we should probably switch to upstream rules as much as possible. But that means lots of testing and stuff and lots of reading rules and see if they are right for Arch. And of course, updating klibc-udev to the same state as udev. I wanted to do it once, then didn't have the time.
Yeah, I know. As far as I'm aware we use as many vanilla rules as possible, with only one smallish file of Arch rules.
On Tue, Aug 26, 2008 at 2:21 AM, Xavier shiningxc@gmail.com wrote:
Hello,
I rewrote stonecrest's python script which generated the Integrity Checks : http://projects.archlinux.org/?p=dbscripts.git;a=blob_plain;f=cron-jobs/chec... last results : http://www.archlinux.org/pipermail/arch-dev-public/2008-August/007655.html
Main changes include :
- better and safer parsing of PKGBUILDs
(it now relies on a separate bash script which source PKGBUILDs and generate python code, rather than trying to parse bash directly in python)
- much better performance
(thanks to a small python C extension which allows to call alpm_vercmp natively rather than having to fork the vercmp binary thousands of times)
- more accurate dependency and provision handling
(having worked on this part of the pacman code for a while probably helped me a bit here)
time ./check_archlinux.sh
=================== = Integrity Check = ===================
Performing integrity checks... ==> checking mismatches ==> checking archs ==> checking dependencies ==> checking makedepends
Missing Dependencies
archboot --> 'bcm43xx-fwcutter>=006-2'
Missing Makedepends
wpa_supplicant --> 'kernel26<2.6.25' xosd --> 'xmms' flac --> 'xmms'
Summary
Invalid PKGBUILDs: 0 Mismatching PKGBUILD names: 0 Invalid archs: 0 Missing (make)dependencies: 4
./check_archlinux.sh 6,29s user 0,65s system 96% cpu 7,210 total
Awesome work.
I know I didn't respond to your emails just yet, but if you could possibly generate a git patch against the dbscripts repo (on projects.archlinux.org), I would apply it. This looks great.
Thanks for the work!
On Tue, Aug 26, 2008 at 9:21 AM, Xavier shiningxc@gmail.com wrote:
Hello,
I rewrote stonecrest's python script which generated the Integrity Checks : http://projects.archlinux.org/?p=dbscripts.git;a=blob_plain;f=cron-jobs/chec... last results : http://www.archlinux.org/pipermail/arch-dev-public/2008-August/007655.html
Main changes include :
- better and safer parsing of PKGBUILDs
(it now relies on a separate bash script which source PKGBUILDs and generate python code, rather than trying to parse bash directly in python)
- much better performance
(thanks to a small python C extension which allows to call alpm_vercmp natively rather than having to fork the vercmp binary thousands of times)
- more accurate dependency and provision handling
(having worked on this part of the pacman code for a while probably helped me a bit here)
I made one design change. Finally I found that generating python code was a bit hackish and less flexible, so I reverted back to a DESC-like format, very easy to parse in python. And this is the namcap way : the python script just calls the bash parsing script then read and parse its output. I finished to implement all features the original script had, mostly the repo hierarchy one and circular deps checking. About circular deps, the old algorithm used was a bit messy and hard to understand, so I just chose to implement an existing one : http://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algori... This resulted in better and complete results, and also at a much better performance (old one took ~20 seconds in cygwin, new one is instant).
The only feature that I refused to keep is the "Missing Repo Packages" one which checked the package existence on ftp.archlinux.org . This was the only one using network, and in my opinion, it does not have this place in this script. Instead, someone knowing the layout of the arch server could write another very simple script checking that all packages in the abs tree have a respective package in the package dir, if both happen to be on the same server.
Enough with the technical details, here are the stripped results :
1) Results for core and extra
$ ./check_packages.py --abs-tree=/var/abs --repos=core,extra
Missing Dependencies ---------------------- archboot --> 'bcm43xx-fwcutter>=006-2'
Missing Makedepends --------------------- wpa_supplicant --> 'kernel26<2.6.25' xosd --> 'xmms' flac --> 'xmms'
Repo Hierarchy for Makedepends -------------------------------- core/iputils depends on extra/jade core/madwifi-utils depends on extra/sharutils core/e2fsprogs depends on extra/bc core/ca-certificates depends on extra/ruby core/madwifi depends on extra/sharutils
Circular Dependencies ----------------------- glibc>bash>readline>ncurses>glibc db>coreutils>shadow>pam>db eclipse-ecj>gcc-gcj>eclipse-ecj
2) Results for community
$ ./check_packages.py --abs-tree=/var/abs --repos=core,extra,community --exclude=core,extra
Missing PKGBUILDs ------------------- community/kde/kdestyle-lipstik
Missing Dependencies ---------------------- flumotion --> 'twisted-web' qc-usb --> 'kernel26<2.6.26' gg2 --> 'arts' eclipse-ve --> 'eclipse<3.3' man-pages-cs --> 'groff-utf8'
Missing Makedepends --------------------- gensplash --> 'klibc-beyond' pygoocanvas --> 'pygobject-doc'
On Thu, Aug 28, 2008 at 6:30 PM, Xavier shiningxc@gmail.com wrote:
Missing Makedepends
xosd --> 'xmms' flac --> 'xmms'
Repo Hierarchy for Makedepends
core/iputils depends on extra/jade core/madwifi-utils depends on extra/sharutils core/e2fsprogs depends on extra/bc core/ca-certificates depends on extra/ruby core/madwifi depends on extra/sharutils
I just read one comment from Thomas : http://bugs.archlinux.org/task/6841#comment32155
Is it fine for core package to have makedepends in extra? What about extra and community (the xmms makedepends above)? The second would mean core and extra repos can't be checked properly on their own..
Circular Dependencies
glibc>bash>readline>ncurses>glibc db>coreutils>shadow>pam>db eclipse-ecj>gcc-gcj>eclipse-ecj
What about this, are cycles accepted too? There is also a bug report for that : FS#9177 Especially the glibc->bash dep does not make sense. For scriptlet to work properly, you need to have a base system installed anyway .. I don't know enough about the other packages to tell which other deps are questionable.
On Fri, Aug 29, 2008 at 4:01 PM, Xavier shiningxc@gmail.com wrote:
I just read one comment from Thomas : http://bugs.archlinux.org/task/6841#comment32155
Is it fine for core package to have makedepends in extra? What about extra and community (the xmms makedepends above)? The second would mean core and extra repos can't be checked properly on their own..
Another related BR is http://bugs.archlinux.org/task/10930, which Roman closed this morning.
So this in fact means that official repos are hierarchical: core -> extra -> community. I have to make clear I have no problems with it, I just want a confirmation. As far as I knew the difference between extra and community (excluding the obvious package popularity) was only in the maintainers. If this is true, then there's no problem for cross dependencies between extra and community, but it seems I was wrong.
My opinion is that core should be completely independent from the rest. Extra and community, au contraire, shouldn't have this constraint., they are not meant to be a basic package set. I don't l'ike cross dependencies, they force a *user* to enable a different repo, but I think cross makedepends are fine, since they only affect packagers. See the BR I linked above... pulseaudio's users won't be able to use native audacious support because the packager didn't have it installed. It is our duty to provide as many features as possible (as long as they're included in the vanilla package), and pulseaudio defaults as enabled in audacious, but isn't built in our package. That's why I think makedepends are fine, otherwise some TU's work will look like it's broken while it's not.
Corrado
On Fri, 29 Aug 2008, bardo wrote:
On Fri, Aug 29, 2008 at 4:01 PM, Xavier shiningxc@gmail.com wrote:
I just read one comment from Thomas : http://bugs.archlinux.org/task/6841#comment32155
Is it fine for core package to have makedepends in extra? What about extra and community (the xmms makedepends above)? The second would mean core and extra repos can't be checked properly on their own..
Another related BR is http://bugs.archlinux.org/task/10930, which Roman closed this morning.
So this in fact means that official repos are hierarchical: core -> extra -> community. I have to make clear I have no problems with it, I just want a confirmation. As far as I knew the difference between extra and community (excluding the obvious package popularity) was only in the maintainers. If this is true, then there's no problem for cross dependencies between extra and community, but it seems I was wrong.
My opinion is that core should be completely independent from the rest. Extra and community, au contraire, shouldn't have this constraint., they are not meant to be a basic package set. I don't l'ike cross dependencies, they force a *user* to enable a different repo, but I think cross makedepends are fine, since they only affect packagers. See the BR I linked above... pulseaudio's users won't be able to use native audacious support because the packager didn't have it installed. It is our duty to provide as many features as possible (as long as they're included in the vanilla package), and pulseaudio defaults as enabled in audacious, but isn't built in our package. That's why I think makedepends are fine, otherwise some TU's work will look like it's broken while it's not.
Corrado
Packages in extra shouldn't makedepends or depends on packages from community. Xmms is an exception. The person who initially moved it to community didn't checked if it was required by other packages in extra. As I maintain xmms in community and am a dev, we decided not to bother moving xmms back in extra. Plus, it is only required as a makedepends. At least, that's the current state of things.
On Fri, Aug 29, 2008 at 5:55 PM, Eric Belanger belanger@astro.umontreal.ca wrote:
Packages in extra shouldn't makedepends or depends on packages from community. Xmms is an exception. The person who initially moved it to community didn't checked if it was required by other packages in extra. As I maintain xmms in community and am a dev, we decided not to bother moving xmms back in extra. Plus, it is only required as a makedepends. At least, that's the current state of things.
Then we could "suggest" official maintainers to install packages that are needed for full feature with a comment inside build(). Otherwise offending packages should be documented somewhere, because there's no hint anywhere that such problems exist.
The only "automagic" alternative I see is the addition of a crossdeps field in PKGBUILDs. What could it do? A lot of things... notify the user at installation time. Notify the maintainer at compile time. Check if the packager has a "buildcrossdeps" option set in makepkg.conf and, if he does, treat them like makedepends. But I don't know. It is a bit overkill for our situation, but other pacman-based distros that have a different repo structure could make great use of it.
Corrado
2008/8/29 Xavier shiningxc@gmail.com:
On Thu, Aug 28, 2008 at 6:30 PM, Xavier shiningxc@gmail.com wrote:
Missing Makedepends
xosd --> 'xmms' flac --> 'xmms'
Repo Hierarchy for Makedepends
core/iputils depends on extra/jade core/madwifi-utils depends on extra/sharutils core/e2fsprogs depends on extra/bc core/ca-certificates depends on extra/ruby core/madwifi depends on extra/sharutils
While I think it's generally better to have makedepends in the same repo - I think these packages should not be moved into Core just because they are makedepends, because we will pollute Core with packages that don't fit there.
participants (11)
-
Aaron Griffin
-
Amanai
-
bardo
-
Bjørn Hamra
-
Eric Belanger
-
from Saanichton BC
-
Gustavo A. Gómez Farhat
-
Roman Kyrylych
-
Thomas Bächler
-
Xavier
-
甘露(Lu Gan)