[aur-general] Packages in Community and votes.

Loui Chang

10 Nov 2008 10 Nov '08

2:59 p.m.

On Mon, Nov 10, 2008 at 07:20:22AM -0700, w9ya wrote:

I would like to remind everyone that the TU system was NOT originated solely to be based on votes, in fact there was no voting until much more recently.

Also, when the voting was added to the TU system the community and TUs were SPECIFICALLY told that the voting system would NEVER be used to make decisions about or demands concerning what any individual TU decided to add. There WERE concerns that just this type of "accounting" would be used to determine how or what a TU may do. ONLY the TUs themselves make these decisions. It has ALWAYS been that way.

BTW.... I am not sure why this happens every few months or so, but it is a repeating thought that somehow the voting system is a milestone setter and thereby an issue for the TUs. If there is something in the wiki or other documentation that would suggest or even say as much IT NEEDS TO BE REMOVED.

Guys, if this is not clear to you, a search of the older mails should yield a wealth of information about this from previous outbursts concerning the suggested requirments for TUs from the voting system results.

This thread diverged from: [arch-dev-public] pkgstats: first results If you have packages in community with zero or one votes there is a problem. You are abusing the resources of community plain and simple. Your packages should definitely have one vote (yourself). If you can't even garner one other person to vote there's not much point in putting the package into community. You'll upload it to the server, it will cost diskspace and bandwidth as it propagates through the mirrors, and no one will use it. Of course votes are not an absolute indicator of usage, but they are somewhat accurate as pkgstats shows. This means we need to change the way community is managed so we can make more efficient use of our resources. I've created a wiki page to outline my ideas: http://wiki.archlinux.org/index.php/Community Another point to consider is that gerolde (the Arch Linux server) is already being taxed to the point where it is becoming unstable. We can either try to do something to help the situation, or we can protect some maintainer's pride of having 100 unused packages in community.

Show replies by date

Xavier

10 Nov 10 Nov

3:12 p.m.

On Mon, Nov 10, 2008 at 3:59 PM, Loui Chang wrote:

...

Another point to consider is that gerolde (the Arch Linux server) is already being taxed to the point where it is becoming unstable. We can either try to do something to help the situation, or we can protect some maintainer's pride of having 100 unused packages in community.

On the other hand, the unpopular packages are probably not be the main responsible of resource problem, are they? I don't know much about this topic, so I could be wrong, but it is interesting to discuss anyway :)

Loui Chang

3:15 p.m.

On Mon, Nov 10, 2008 at 04:12:14PM +0100, Xavier wrote:

...

On Mon, Nov 10, 2008 at 3:59 PM, Loui Chang wrote:

...
Another point to consider is that gerolde (the Arch Linux server) is already being taxed to the point where it is becoming unstable. We can either try to do something to help the situation, or we can protect some maintainer's pride of having 100 unused packages in community.

On the other hand, the unpopular packages are probably not be the main responsible of resource problem, are they? I don't know much about this topic, so I could be wrong, but it is interesting to discuss anyway :)

If you have 300 of them sure they will.

Firmicus

3:15 p.m.

Loui Chang a écrit :

...

On Mon, Nov 10, 2008 at 07:20:22AM -0700, w9ya wrote:

...
I would like to remind everyone that the TU system was NOT originated solely to be based on votes, in fact there was no voting until much more recently.

Also, when the voting was added to the TU system the community and TUs were SPECIFICALLY told that the voting system would NEVER be used to make decisions about or demands concerning what any individual TU decided to add. There WERE concerns that just this type of "accounting" would be used to determine how or what a TU may do. ONLY the TUs themselves make these decisions. It has ALWAYS been that way.

BTW.... I am not sure why this happens every few months or so, but it is a repeating thought that somehow the voting system is a milestone setter and thereby an issue for the TUs. If there is something in the wiki or other documentation that would suggest or even say as much IT NEEDS TO BE REMOVED.

Guys, if this is not clear to you, a search of the older mails should yield a wealth of information about this from previous outbursts concerning the suggested requirments for TUs from the voting system results.

This thread diverged from: [arch-dev-public] pkgstats: first results

If you have packages in community with zero or one votes there is a problem. You are abusing the resources of community plain and simple.

Your packages should definitely have one vote (yourself). If you can't even garner one other person to vote there's not much point in putting the package into community. You'll upload it to the server, it will cost diskspace and bandwidth as it propagates through the mirrors, and no one will use it.

Of course votes are not an absolute indicator of usage, but they are somewhat accurate as pkgstats shows.

This means we need to change the way community is managed so we can make more efficient use of our resources.

I've created a wiki page to outline my ideas: http://wiki.archlinux.org/index.php/Community

Another point to consider is that gerolde (the Arch Linux server) is already being taxed to the point where it is becoming unstable. We can either try to do something to help the situation, or we can protect some maintainer's pride of having 100 unused packages in community.

OK, but you're going a bit too fast, Loui. Please give us some time to organize the next TU Meeting first, where this and other issues will be discussed openly. On the arch-dev list it was explicitly mentioned that this is something that should be discussed among Trusted Users first. Besides, many TUs have not yet been properly informed of the background to this discussion. Thanks for your patience, F

w9ya

4:31 p.m.

Sigh.... I do NOT normally comment here and certainly NOT with the tone I am about to use. But you guys (plural) are attempting to dictate binding discussions without first doing your own due diligence. Or you are asking me and others to do your due diligence for you? This is, of course, very troubling to me. It should be to other TUs too. It is NOT a given that the voting statistics are accurate or even "..somewhat accurate". MANY reasons have been given in the past why such accuracy is not possible under the current voting scheme, so I again ask you to do your due diligence, i.e. PLEASE, PLEASE, PLEASE do a search on previous discussions about this that have taken place (repeatedly) over (at least) the last couple of years. This insistence to NOT do this searching on previous discussions before issuing your **opinions**, and declaring them as having some form of accuracy (facts) is troubling !! I know for a *FACT* that many of my packages are used, some VERY heavily and by MANY users and yet have NO votes. I will give you here but two of the several reasons; 1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download packages or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes. 2 - Since the votes are NOT reflective of downloads, and for the above reason downloads are NOT reflective of the numbers of users, and FURTHER many users do NOT vote, there can be NO correlation between votes and usage. It isn't even a rough estimate. Best regards; Bob Finch P.S... If the problem is that the server's output needs to be improved, then that should be the focus of the discussion from the very beginning LONG BEFORE we discuss ANY specific options (voting restrictions). If that was why this topic came up again, thank you for at least announcing it now instead of after we spent more time on this voting=server space. (And server space needs not, and should not be affecting it becoming "bogged down" as that usually is a throughput issue. On Mon, Nov 10, 2008 at 7:59 AM, Loui Chang wrote:

...

On Mon, Nov 10, 2008 at 07:20:22AM -0700, w9ya wrote:

...
I would like to remind everyone that the TU system was NOT originated solely to be based on votes, in fact there was no voting until much more recently.

Also, when the voting was added to the TU system the community and TUs were SPECIFICALLY told that the voting system would NEVER be used to make decisions about or demands concerning what any individual TU decided to add. There WERE concerns that just this type of "accounting" would be used to determine how or what a TU may do. ONLY the TUs themselves make these decisions. It has ALWAYS been that way.

BTW.... I am not sure why this happens every few months or so, but it is a repeating thought that somehow the voting system is a milestone setter and thereby an issue for the TUs. If there is something in the wiki or other documentation that would suggest or even say as much IT NEEDS TO BE REMOVED.

Guys, if this is not clear to you, a search of the older mails should yield a wealth of information about this from previous outbursts concerning the suggested requirments for TUs from the voting system results.

This thread diverged from: [arch-dev-public] pkgstats: first results

If you have packages in community with zero or one votes there is a problem. You are abusing the resources of community plain and simple.

Your packages should definitely have one vote (yourself). If you can't even garner one other person to vote there's not much point in putting the package into community. You'll upload it to the server, it will cost diskspace and bandwidth as it propagates through the mirrors, and no one will use it.

Of course votes are not an absolute indicator of usage, but they are somewhat accurate as pkgstats shows.

This means we need to change the way community is managed so we can make more efficient use of our resources.

I've created a wiki page to outline my ideas: http://wiki.archlinux.org/index.php/Community

Another point to consider is that gerolde (the Arch Linux server) is already being taxed to the point where it is becoming unstable. We can either try to do something to help the situation, or we can protect some maintainer's pride of having 100 unused packages in community.

Loui Chang

5:19 p.m.

On Mon, Nov 10, 2008 at 09:31:17AM -0700, w9ya wrote:

...

I do NOT normally comment here and certainly NOT with the tone I am about to use. But you guys (plural) are attempting to dictate binding discussions without first doing your own due diligence. Or you are asking me and others to do your due diligence for you? This is, of course, very troubling to me. It should be to other TUs too.

Past discussions are less relevant because they don't have the same effects as the problems we face today.

...

It is NOT a given that the voting statistics are accurate or even "..somewhat accurate". MANY reasons have been given in the past why such accuracy is not possible under the current voting scheme, so I again ask you to do your due diligence, i.e. PLEASE, PLEASE, PLEASE do a search on previous discussions about this that have taken place (repeatedly) over (at least) the last couple of years.

Nothing in life is a given, but we make conclusions based in empirical data. This is how decisions are made to improve things. It's what drives advancement.

...

I know for a *FACT* that many of my packages are used, some VERY heavily and by MANY users and yet have NO votes. I will give you here but two of the several reasons;

...

1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download packages or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes.

If your community cares about the packages that are provided in [community] they should vote. Voting wasn't put in the AUR for absolutely no reason. If someone doesn't vote for a package then I would assume that it isn't all that important if the package is in community or not. If your users don't even download the packages then there is no point for those packages to be in community.

...

2 - Since the votes are NOT reflective of downloads, and for the above reason downloads are NOT reflective of the numbers of users, and FURTHER many users do NOT vote, there can be NO correlation between votes and usage. It isn't even a rough estimate.

It is a rough estimate. Please review the pkgstats results. Furthermore you're making it sound like moving a package from community is some kind of travesty, like it will disappear. No. You can still maintain it in unsupported, and you can still run your own repo like the fine folks running the arch-games repo. Also, maybe you should recommend server upgrades to your offshoot distro so you're able to host your own niche repo instead of telling Arch to serve people who don't even want to participate in the community.

w9ya

5:54 p.m.

On Mon, Nov 10, 2008 at 10:19 AM, Loui Chang wrote:

...

On Mon, Nov 10, 2008 at 09:31:17AM -0700, w9ya wrote:

...
I do NOT normally comment here and certainly NOT with the tone I am about to use. But you guys (plural) are attempting to dictate binding discussions without first doing your own due diligence. Or you are asking me and others to do your due diligence for you? This is, of course, very troubling to me. It should be to other TUs too.

Past discussions are less relevant because they don't have the same effects as the problems we face today.

An assumption on your part. I doubt it is true either IF the problem is server overload of some kind. Arch has often had those problems in the past, and IN FACT it was a strain at first to accomodate the user repos.

...

...
It is NOT a given that the voting statistics are accurate or even "..somewhat accurate". MANY reasons have been given in the past why such accuracy is not possible under the current voting scheme, so I again ask you to do your due diligence, i.e. PLEASE, PLEASE, PLEASE do a search on previous discussions about this that have taken place (repeatedly) over (at least) the last couple of years.

Nothing in life is a given, but we make conclusions based in empirical data. This is how decisions are made to improve things. It's what drives advancement.

But this data is NOT empirical at all. To say as much is to ignore what the methods arr and how it is collected and from what percentage of the whole and so forth.

...

...
I know for a *FACT* that many of my packages are used, some VERY heavily and by MANY users and yet have NO votes. I will give you here but two of the several reasons;

...
1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download packages or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes.

If your community cares about the packages that are provided in [community] they should vote. Voting wasn't put in the AUR for absolutely no reason. If someone doesn't vote for a package then I would assume that it isn't all that important if the package is in community or not.

Well, again another history lesson for ya; It was added ONLY as a guide becuase some TUs wanted to knwo at least something. It was WELL understood that it was a lousy gauge of ANYTHING specific and could NEVER be empirically used.

...

If your users don't even download the packages then there is no point for those packages to be in community.

Well if fact I get comments about them from direct arhc linux users, so that only goes to point out how lousy the voting system is if it is suppose to say if the progrma is being used.

...

...
2 - Since the votes are NOT reflective of downloads, and for the above reason downloads are NOT reflective of the numbers of users, and FURTHER many users do NOT vote, there can be NO correlation between votes and usage. It isn't even a rough estimate.

It is a rough estimate. Please review the pkgstats results.

Rough is an interesting word. It covers a lot of ground. So, above you said it meant something empirical. Exactly what does a rough estimate mean, and how does that meaning relate to it being empirical ?

...

Furthermore you're making it sound like moving a package from community is some kind of travesty, like it will disappear.

No. You can still maintain it in unsupported, and you can still run your own repo like the fine folks running the arch-games repo.

Also, maybe you should recommend server upgrades to your offshoot distro so you're able to host your own niche repo instead of telling Arch to serve people who don't even want to participate in the community.

Again, you do NOT understand what the community system is or what it was

suppose to accomplish. Please read some of the earlier email threads on this. I sincerely ask you to educate yourself about the reasons this thing has evolved to the system it is currently. It is NOT merely a bunch of repos with users deciding what is to become a binary package. If it was, that would be so much LESS than what it offers now. And it arch would become just another distro, no better or worse, merely different. The excellent wiki usage AND the excellent free-flowing community system are what make arch uniquely better. And good participation in a wiki can be had in other ways, but the community system is truly a wonderful free-market style system. What you suggest is a restriction for want of a better server support. IMnsHO, this will be a bad reason to do the kinds of things you suggest. Best regards; Bob Finch

Aaron Griffin

5:21 p.m.

On Mon, Nov 10, 2008 at 10:31 AM, w9ya wrote:

...

1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download packages or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes.

Just to point out a flaw here. Users of these packages are not ArchLinux users. They are shackstick users. So votes don't make sense for ArchLinux. These packages would not make sense in an ArchLinux repo, on an ArchLinux server, for use by 1 or 2 ArchLinux users, and a few hundred users of another distribution. So, in the case you have outlined, the voting is working perfectly as intended.

Angel Velásquez

5:47 p.m.

On Mon, Nov 10, 2008 at 6:21 PM, Aaron Griffin wrote:

...

On Mon, Nov 10, 2008 at 10:31 AM, w9ya wrote:

...
1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download packages or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes.

Just to point out a flaw here. Users of these packages are not ArchLinux users. They are shackstick users. So votes don't make sense for ArchLinux. These packages would not make sense in an ArchLinux repo, on an ArchLinux server, for use by 1 or 2 ArchLinux users, and a few hundred users of another distribution. So, in the case you have outlined, the voting is working perfectly as intended.

The explain that bfinch shows is not the case, as I said (second time this day), there is an amount of packages (aspell,i18n related) which break the rule about "votes needed". I mean, maybe there is not chinesse support in Arch Linux, but why if we got a TU or Dev who speaks chinesse and he wants to move chinesse language packages to community?, he won't be able cause the packages aren't enough voted? this is very unfair, and that's why I think votes isn't the unique point to focus when a package is moved to community. I understand the fact about moving -non-popular or unuseful- packages to community and waste resources, is bad, and I know that exists hundres of packages without votes, IMHO the correct way to handle this is simply, the TU who add packages with very few votes should give a good reason about why he did it, and in case other TUs aren't agree the TU who upload the package should find better reasons, and try later. And again *language packages break the rule* simply. Thanks -- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909 Arch Linux Trusted User

Loui Chang

5:53 p.m.

On Mon, Nov 10, 2008 at 06:47:32PM +0100, Angel Velásquez wrote:

...

On Mon, Nov 10, 2008 at 6:21 PM, Aaron Griffin wrote:

...
On Mon, Nov 10, 2008 at 10:31 AM, w9ya wrote:

...
1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download packages or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes.

Just to point out a flaw here. Users of these packages are not ArchLinux users. They are shackstick users. So votes don't make sense for ArchLinux. These packages would not make sense in an ArchLinux repo, on an ArchLinux server, for use by 1 or 2 ArchLinux users, and a few hundred users of another distribution. So, in the case you have outlined, the voting is working perfectly as intended.

The explain that bfinch shows is not the case, as I said (second time this day), there is an amount of packages (aspell,i18n related) which break the rule about "votes needed". I mean, maybe there is not chinesse support in Arch Linux, but why if we got a TU or Dev who speaks chinesse and he wants to move chinesse language packages to community?, he won't be able cause the packages aren't enough voted? this is very unfair, and that's why I think votes isn't the unique point to focus when a package is moved to community. I understand the fact about moving -non-popular or unuseful- packages to community and waste resources, is bad, and I know that exists hundres of packages without votes, IMHO the correct way to handle this is simply, the TU who add packages with very few votes should give a good reason about why he did it, and in case other TUs aren't agree the TU who upload the package should find better reasons, and try later. And again *language packages break the rule* simply.

Thanks

Yeah I agree there should be room for some exceptions. Dependencies would be the obvious exceptions, and maybe perhaps i18n packages should be included as well (optdepends?).

Loui Chang

5:59 p.m.

On Mon, Nov 10, 2008 at 12:53:13PM -0500, Loui Chang wrote:

...

On Mon, Nov 10, 2008 at 06:47:32PM +0100, Angel Velásquez wrote:

...
The explain that bfinch shows is not the case, as I said (second time this day), there is an amount of packages (aspell,i18n related) which break the rule about "votes needed". I mean, maybe there is not chinesse support in Arch Linux, but why if we got a TU or Dev who speaks chinesse and he wants to move chinesse language packages to community?, he won't be able cause the packages aren't enough voted? this is very unfair, and that's why I think votes isn't the unique point to focus when a package is moved to community. I understand the fact about moving -non-popular or unuseful- packages to community and waste resources, is bad, and I know that exists hundres of packages without votes, IMHO the correct way to handle this is simply, the TU who add packages with very few votes should give a good reason about why he did it, and in case other TUs aren't agree the TU who upload the package should find better reasons, and try later. And again *language packages break the rule* simply.

Thanks

Yeah I agree there should be room for some exceptions. Dependencies would be the obvious exceptions, and maybe perhaps i18n packages should be included as well (optdepends?).

I've included the exception for i18n packages in: http://wiki.archlinux.org/index.php/Community

Alessio Bolognino

6:18 p.m.

On Mon 2008-11-10 12:59, Loui Chang wrote:

...

On Mon, Nov 10, 2008 at 12:53:13PM -0500, Loui Chang wrote:

...
On Mon, Nov 10, 2008 at 06:47:32PM +0100, Angel Velásquez wrote: [...]

Yeah I agree there should be room for some exceptions. Dependencies would be the obvious exceptions, and maybe perhaps i18n packages should be included as well (optdepends?).

I've included the exception for i18n packages in: http://wiki.archlinux.org/index.php/Community

What about perl-* packages? IIRC there is an automagic script that mirrors CPAN. -- Alessio (molok) Bolognino

Loui Chang

6:27 p.m.

On Mon, Nov 10, 2008 at 07:18:51PM +0100, Alessio Bolognino wrote:

...

On Mon 2008-11-10 12:59, Loui Chang wrote:

...
On Mon, Nov 10, 2008 at 12:53:13PM -0500, Loui Chang wrote:

...
On Mon, Nov 10, 2008 at 06:47:32PM +0100, Angel Velásquez wrote: [...]

Yeah I agree there should be room for some exceptions. Dependencies would be the obvious exceptions, and maybe perhaps i18n packages should be included as well (optdepends?).

I've included the exception for i18n packages in: http://wiki.archlinux.org/index.php/Community

What about perl-* packages? IIRC there is an automagic script that mirrors CPAN.

Do you think they should really be part of community though? Wouldn't it be sufficient for them to be in unsupported?

Abhishek Dasgupta

6:32 p.m.

2008/11/10 Loui Chang :

...

Do you think they should really be part of community though? Wouldn't it be sufficient for them to be in unsupported?

If the packages can be (reliably) automatically built and maintained, unless there's a resource problem, they can be easily part of [community]; otherwise unsupported is fine as well... -- Abhishek Dasgupta

Firmicus

8:12 p.m.

Loui Chang a écrit :

...

On Mon, Nov 10, 2008 at 07:18:51PM +0100, Alessio Bolognino wrote:

...
On Mon 2008-11-10 12:59, Loui Chang wrote:

...
On Mon, Nov 10, 2008 at 12:53:13PM -0500, Loui Chang wrote:

...
On Mon, Nov 10, 2008 at 06:47:32PM +0100, Angel Velásquez wrote: [...]

Yeah I agree there should be room for some exceptions. Dependencies would be the obvious exceptions, and maybe perhaps i18n packages should be included as well (optdepends?).

I've included the exception for i18n packages in: http://wiki.archlinux.org/index.php/Community

What about perl-* packages? IIRC there is an automagic script that mirrors CPAN.

Yes. perl-cpanplus-pacman (in community) which I wrote a while back.

...

Do you think they should really be part of community though? Wouldn't it be sufficient for them to be in unsupported?

Absolutely. And I maintain a few of these myself that I picked up from the perl mess that some former TU (xterminus I think) had abandoned in community in the summer of 2007. I have moved of LOT of them to unsupported but many are still in community. Now a few days ago (that is, before that discussion started) I looked at the perl packages I currently maintain in community, and identified over 75 that are good candidates to be moved to unsupported, because they have very few votes. I will check this against the pkgstats value and will make a cleanup myself in the next few days/weeks. In a different matter, we have had a discussion last year about a possible limit to the number of packages each TU should be allowed to maintain in community, but we could not reach any agreement on this. Perhaps time is ripe to discuss this issue once more. F

Daenyth Blank

10:32 p.m.

...

In a different matter, we have had a discussion last year about a possible limit to the number of packages each TU should be allowed to maintain in community, but we could not reach any agreement on this. Perhaps time is ripe to discuss this issue once more. I'd rather try to keep discussion points separate. FWIW though I am against any hard and fast rules for this. We can talk it over though.

I'd like anyone with further points to add them to here: http://wiki.archlinux.org/index.php/Talk:Community#Other_discussion_topics This is so that we can better structure the upcoming TU meeting. I'll work on sorting things out as I go. On that note, what are the exact problems we face now? I would say that there are at least 2, though they're closely related: 1) The server is being strained (what parts exactly?) by the community repo. 2) TUs don't seem to be paying attention to the community's need for packages they upload, which wastes resources.

Aaron Griffin

11:05 p.m.

On Mon, Nov 10, 2008 at 4:32 PM, Daenyth Blank wrote:

...

1) The server is being strained (what parts exactly?) by the community repo.

It's primarily disk space and IO load issues. The community repo and AUR are fairly large. A cron job which WAS keeping the AUR's permissions in check was actually pegging our system with so much load that we had to remove any handling of the AUR files (hope the code is good enough for that). The AUR backend daemon opens every single package file (wtf?) when it runs, which is a HUGE resource hog. In an ideal world, someone would rewrite this to work the same way the offical repos work - with repo-add and a separate decoupled script to load the mysql database from a pacman DB. Sizes on disk: community: 11G extra: 11G core: 330M unsupported: 800M

Aaron Schaefer

11 Nov 11 Nov

12:22 a.m.

...

It's primarily disk space and IO load issues.

I have questions, mostly meant to get people thinking about alternatives and ramifications of any solutions... If the real issue is disk space and IO, what about the possibility of a hardware upgrade? What about moving the largest packages to unsupported (or something like arch-games) instead of basing it on votes? It looks like eliminating just the top 10 largest community packages would save 1.8 GB of space! See http://rafb.net/p/Xfw0gh39.html for package sizes. What about putting community on it's own server? What about fixing the AUR backend? What about adding a CVS commit hook in the mean time to fix permissions on upload instead of running a single cron job? If we make these proposed changes, how will they actually impact the server and it's current problems? How will they effect Arch users? What is the price of convenience that the community repo provides to Arch users? Will there be a way to easily differentiate packages in unsupported that are actually maintained by TUs? How can we reliably tell what is popular? Download numbers, voting, pkgstats, etc. all have their own issues and biases...is there a better way? What makes the most sense in the long run when there are sure to be more TUs and packages in community eventually? Should we worry about things that are currently in community, or just new packages? My main point is that there are many options, and any solution that gets acted upon needs to be based on hard evidence for improvement and account for all consequences of that change rather than just basing it on what sounds good. There has been a lot of rabble-rousing and not much investigation into the underlying problems and proposed solutions. -- Aaron "ElasticDog" Schaefer

Loui Chang

1 a.m.

On Mon, Nov 10, 2008 at 07:22:01PM -0500, Aaron Schaefer wrote:

...

...
It's primarily disk space and IO load issues.

I have questions, mostly meant to get people thinking about alternatives and ramifications of any solutions...

If the real issue is disk space and IO, what about the possibility of a hardware upgrade? What about moving the largest packages to unsupported (or something like arch-games) instead of basing it on votes? It looks like eliminating just the top 10 largest community packages would save 1.8 GB of space! See http://rafb.net/p/Xfw0gh39.html for package sizes. What about putting community on it's own server? What about fixing the AUR backend? What about adding a CVS commit hook in the mean time to fix permissions on upload instead of running a single cron job?

If we make these proposed changes, how will they actually impact the server and it's current problems? How will they effect Arch users? What is the price of convenience that the community repo provides to Arch users? Will there be a way to easily differentiate packages in unsupported that are actually maintained by TUs? How can we reliably tell what is popular? Download numbers, voting, pkgstats, etc. all have their own issues and biases...is there a better way? What makes the most sense in the long run when there are sure to be more TUs and packages in community eventually? Should we worry about things that are currently in community, or just new packages?

My main point is that there are many options, and any solution that gets acted upon needs to be based on hard evidence for improvement and account for all consequences of that change rather than just basing it on what sounds good. There has been a lot of rabble-rousing and not much investigation into the underlying problems and proposed solutions.

I've said this already in discussions but I'll say this again. Fixing the community back end, removing large packages, and removing unused packages are all possible solutions to the problem. If we implement all the solutions, then we get an incremental improvment. Each solution will build upon the others. We shouldn't only implement one measure. We should implement ALL measures within reason. I only raised the issue of unused or barely used packages in Community and pruning the repo. We should really be focusing on that before diverting the discussion and delving into other areas. http://wiki.archlinux.org/index.php/Community

Aaron Schaefer

1:39 a.m.

On Mon, Nov 10, 2008 at 8:00 PM, Loui Chang wrote:

...

I only raised the issue of unused or barely used packages in Community and pruning the repo. We should really be focusing on that before diverting the discussion and delving into other areas.

I know we've discussed this in IRC, but again my question is _why_ should this be the focus? Does it give us the most improvement for the least effort? Does it inconvenience users the least? Is it the cheapest? By how much? Is it the fastest? Why? Why? Why? I'm open-minded about suggestions, but need something more substantial to back them up than just saying "we should do this". Where are the numbers to support the claim? Also, it seems as though the issue of popularity/voting and the community repo might be altogether different than the issue of server resources. Are we linking the two together because of a twist of fate with timing and pkgstats coming to fruition? Aaron "ElasticDog" Schaefer p.s. my link for package sizes will disappear in a day, here's a better one: http://omploader.org/vd3Vx --

Aaron Griffin

3:50 p.m.

On Mon, Nov 10, 2008 at 6:22 PM, Aaron Schaefer wrote:

...

If the real issue is disk space and IO, what about the possibility of a hardware upgrade?

Insider info: we're upgrading from ram and disks next Thursday, I believe, and switching to x86_64 (as we'll have 16GB of ram). The disk space won't see as large of an improvement as we'd like, due to hardware compatibility stuff (damn Dell). This will help the space issue. The space issue is the primary reason I haven't flipped to switch to generate source tarballs yet (last I checked, this was 10-15 gigs for core and extra). And with us distributing community packages, I will need too, legally, provide source for all those binaries too... The IO, on the other hand

...

What about fixing the AUR backend? What about adding a CVS commit hook in the mean time to fix permissions on upload instead of running a single cron job?

is related to the AUR backend daemon (tupkgupdate ?), which is in the AUR repository. A rewrite would be HUGELY appreciated.... the steps are simple: find new packages in upload dir, add to pacman DB, add to mysql DB. It's not all that hard, yet the current daemon runs through ALL packages when it does this, IIRC

Abhishek Dasgupta

10 Nov 10 Nov

6:03 p.m.

pkgstats results are far more reliable than votes, especially if a package has been added to [community] when it had less votes and has since become popular. Many people will find the package in the repository and simply download it. They may not even know (at first) about the AUR, or about the voting system. Also, since Arch is a relatively small distribution, quite a few packages which are otherwise used (and important) are not in binary form in Arch, because too few people use them. The problem in adding packages with zero (or <5 votes) is not only of resources, but also of what happens when the TU leaves... In these cases others have to either take a package which they don't use, or move it to unsupported. -- Abhishek Dasgupta

w9ya

5:58 p.m.

On Mon, Nov 10, 2008 at 10:21 AM, Aaron Griffin wrote:

...

...
1 - I maintain an offshoot version of archlinux, derived from faunos, called "shackstick". It is used and is becoming quite popular amongst the ham radio community. It is packaged as a whole and the user does NOT download

On Mon, Nov 10, 2008 at 10:31 AM, w9ya wrote: packages

...
or even is part of the arch linux community, so NO votes are taken. Yet it uses over 25 of my packages that would seem to otherwise be without votes.

Just to point out a flaw here. Users of these packages are not ArchLinux users. They are shackstick users. So votes don't make sense for ArchLinux. These packages would not make sense in an ArchLinux repo, on an ArchLinux server, for use by 1 or 2 ArchLinux users, and a few hundred users of another distribution. So, in the case you have outlined, the voting is working perfectly as intended.

A valid point IF the voting system was in some form of mandatory issuance. At it is currently constituted however, it is used by exactly how many users ? Since using the voting system requires several steps for the user to go through, and is NOT required, so I think it is a BIG stretch to suggest that the numbers have any significant meaning. Heck do we even (truly) KNOW how many users are voting, do we ? So far, I would doubt it is used by any significant user base !! i.e. Goggle earth is a VERY important program, and if archlinux has 5000 users (Let's say that is the number as I have no idea what it really is...) then am I to believe that only approx. 8 percent of the users have any use for it and therefore the others are NOT using it ? Further do we have ANY idea how many of the packages in the extra repo are being used as a percentage of user base ? <- i.e. Is there another metric being used than we can point to as a basis for statistical analysis ? Anyways, I *know* that with the very little sleep last night I am NOT in the best of moods, and you guys are getting the brunt of that. *** However, we should all endeavor to remember that the imperatives of the TU system were specifically designed to be ENTIRELY different than those of the dev-based work. As such, I hope we all can at least agree that the TU system merely becomes more like a chore than the fun thing it was designed to be because (after all,) it would ONLY be a dev-light training ground instead of the all encompassing environment that archlinux has perfected and NO OTHER distro can say it has. And thereby NO OTHER distro can point to the success of it's user community contributing like arch can say with a well earned pride. i.e. Users can work at many different levels, goals, work styles, desires and the current TU system should be able to accommodate them !! Really, I am quite serious about this, NO OTHER distro offers this uniquely styled resource. And it was able to be styled because the people that came before and set this up asked what the goals were FIRST, and then were able to properly gauge what solutions might work best. i.e. Please do not seek to solve one problem and with the consequences mess up this source of fun, enjoyment and pride. Put a straight jacket on a problem that might be solved another way is to put unneeded restrictions that WILL have negative consequences on a system that was designed to be encompassing. Best regards; Bob Finch

5645

Age (days ago)

5646

Last active (days ago)

List overview

Download

22 comments

10 participants

participants (10)

Aaron Griffin
Aaron Schaefer
Abhishek Dasgupta
Alessio Bolognino
Angel Velásquez
Daenyth Blank
Firmicus
Loui Chang
w9ya
Xavier