Firmicus wrote:

I basically agree that we need some form of control.

However I have some reservations and comments.

First, I would like to understand the usefulness of discussing and voting on this proposal first, before tackling the (IMO probably more serious issue) of having too many packages with very low popularity / usefulness in our community repo. Perhaps the former issue is easier to deal with? Or could serve as a basis for dealing with the latter one?

Paving the way... Basically stop the problem getting any worse first. Then do the real fixing. This is probably also the least controversial of the two.

Now to the specific points of the proposal:

...

* Only "popular" packages may enter the repo, as defined by 1% usage from pkgstats or 10 votes on the AUR.

Allan, can you develop a bit more what is your criterion for the figures 1% or 10 votes. Why not 5% or 3 votes? (which in my humble opinion would seem to make more sense...)

I just liked the minimum of 1% for community. Ten votes comes from looking at the number of votes best at discriminating packages with more or less than 1% of usage. In fact any number between 10 and 20 does quite well but people told me 20 was really too many...

...

* Any additions not covered by the above criteria must first be proposed on the aur-general mailing list, explaining the reason for the exemption (e.g. renamed package, new package) at which point a general consensus from the TUs will be reached. TUs with large numbers of "non-popular" packages are more likely to be rejected.

What should a "general consensus" look like? If 5 TUs are for and 3 are against and the remaining ones do not reply, do we say yes or no? Can we be more specific on the above point without burdening the procedure with bureaucratic rules?

That is the main thing that needs defined before I call for a vote. I really want others ideas on this. Perhaps agreement of 3 other TUs.

...

Stefan Hussmann wrote:

...

Some thoughts. - If we encourage people to drop packages that are not popular, we should also encourage them to take packages in "usupported" that _are_ popular to "community".

That would be the idea.

I have examined the repos extra, community and unsupported quite carefully, and as a result I do not think there are that many packages in unsupported that really deserve being in community! Possible candidates are

yaourt, etc: but only if wain would become a TU AND if we think it is a good idea to support that very popular tool (not sure personally – also it is only a script, so easy to install).

Remember we do not support automatic building from unsupported

aurup

Do we support AUR uploaders?

nspluginwrapper-flash + dependencies > well, these are no longer required I believe splashy wine-doors songbird

There are license issues with the icons there. Also, last time I checked it would not build from source without patching xulrunner in ways mozilla devs are against. Otherwise I would have it there now!

Daenyth and others have an arch-games unofficial repo with many games in it.

uswsusp customizepkg pactools

A few scripts in that package are now part of pacman (orphan detection) and others are in the official pacman-contrib scripts.

archassistant vlc-plugin fbsplash

Otherwise, I would rather VERY MUCH encourage TUs to have a look at the packages in EXTRA that are currently orphaned, and to contact the devs if you are interested to maintain some of them in community. There are many more important packages there that would deserve to be maintained in community than being orphaned in extra.

That's it for now.

F

Allan

On Mon, Dec 1, 2008 at 10:25, Firmicus wrote:

By the way, in order to preach by example I just moved 54 of my perl packages from community to unsupported. They all had low usage and were not dependencies. I had to keep some others with few votes because they are needed by pkgs maintained by sergej ;)

To make the cleanup operation easier I wrote this little script:

I'd just like to remind people that moving from community erases all package metadata entirely -- there's no real move operation. If your package has a lot of important votes/comments, I would consider waiting until we come to some backend changes which allow easier moving. Louipc is working on this at the moment, so hopefully it shouldn't be too long.

Why is package popularity judged by votes anyway? I never vote. The reason i never vote is because i don't understand why package popularity can't simply be judged by download count. I know there's been some discussion of this topic earlier but as far as i can recall there were no convincing arguments against it. even if some moron decides to download his package more times to increase the chance of adoption(where would the motivation to do this be anyway? if he simply want's to get it in the pacman system for easier maintenance a simple guide to make your own repository and add it to pacman would remove this incurrence in 99% of cases) the TU's and Devs could still choose to not include the package in the repos. This could also largely be avoided by only counting i guess certain ip ranges(i'm not an expert on these things, but i DO know that counting downloads with some level of security is a common occurence on the net) That is if the other suggestion in this thread doesn't get implemented and TU's and devs are actually free to choose which packages they want to adopt. To me - that seems important - because where's the motivation to maintain a package which you doesn't use? Maybe the motivation is to make a better arch? somehow i doubt this one will matter much in the daily lives of the devs and TU's when they have to take time out of their private schedules to do something that is essentially non-work non-paid. So I think motivation is something that should be encouraged so things doesn't seem to much like work.. So i guess what i'm saying is, why create policy around a flawed system? why not try to fix the core of the problem instead of patching it up? -K

On Wed, Dec 3, 2008 at 2:23 PM, Aaron Griffin wrote:

On Wed, Dec 3, 2008 at 3:15 PM, Kristoffer Fossgård wrote:

...

Why is package popularity judged by votes anyway? I never vote. The reason i never vote is because i don't understand why package popularity can't simply be judged by download count. I know there's been some discussion of this topic earlier but as far as i can recall there were no convincing arguments against it. even if some moron decides to download his package more times to increase the chance of adoption(where would the motivation to do this be anyway? if he simply want's to get it in the pacman system for easier maintenance a simple guide to make your own repository and add it to pacman would remove this incurrence in 99% of cases) the TU's and Devs could still choose to not include the package in the repos. This could also largely be avoided by only counting i guess certain ip ranges(i'm not an expert on these things, but i DO know that counting downloads with some level of security is a common occurence on the net)

We have mirrors. Almost 100 of them. Feel free to contact them all, have them write code to count downloads which then sends the stats to us, and then we can implement this.

What you suggest is absolutely not feasible at all.

Quite frankly Aaron, this attitude is not helpful to your case at all. And it leads to worse as we have seen between you and me as of late. No one likes to be rebutted in such a manner. Yes, it is NOT feasible, BUT you can **choose** to say this nicely or coarsely. Bob F. P.S... I am *still* waiting for answers to my questions from yesterday. But please take as much time as you need to answer them.

On Wed, Dec 3, 2008 at 2:58 PM, Aaron Griffin wrote:

On Wed, Dec 3, 2008 at 3:40 PM, w9ya wrote:

...

On Wed, Dec 3, 2008 at 2:23 PM, Aaron Griffin wrote:

...

On Wed, Dec 3, 2008 at 3:15 PM, Kristoffer Fossgård wrote:

...

Why is package popularity judged by votes anyway? I never vote. The reason i never vote is because i don't understand why package popularity can't simply be judged by download count. I know there's been some discussion of this topic earlier but as far as i can recall there were no convincing arguments against it. even if some moron decides to download his package more times to increase the chance of adoption(where would the motivation to do this be anyway? if he simply want's to get it in the pacman system for easier maintenance a simple guide to make your own repository and add it to pacman would remove this incurrence in 99% of cases) the TU's and Devs could still choose to not include the package in the repos. This could also largely be avoided by only counting i guess certain ip ranges(i'm not an expert

on

...

...

these things, but i DO know that counting downloads with some level of security is a common occurence on the net)

We have mirrors. Almost 100 of them. Feel free to contact them all, have them write code to count downloads which then sends the stats to us, and then we can implement this.

What you suggest is absolutely not feasible at all.

Quite frankly Aaron, this attitude is not helpful to your case at all. And it leads to worse as we have seen between you and me as of late. No one likes to be rebutted in such a manner.

Yes, it is NOT feasible, BUT you can **choose** to say this nicely or coarsely.

Kristoffer, I apologize if this sounded harsh, as Mr Finch seemed to interpret it. I did not mean it as such - I meant to say that you were overlooking the fact that we do not have full control over our mirrors and can only track downloads from one out of *many* servers.

A better attempt. Thank you. Bob F.

Hello,

We have mirrors. Almost 100 of them. Feel free to contact them all, have them write code to count downloads which then sends the stats to us, and then we can implement this.

What you suggest is absolutely not feasible at all.

That's too bad, I wanted to suggest counting of downloads too (because I believe that the number downloads of particular version of a package would after a while correlate quite well with the number of users that actually use, i. e. upgrade this package - it should more or less solve the problem of people trying the package and removing it quickly after that that was mentioned). Anyway I've been meaning to contribute with some ideas for the topic for at least four days (since I read the first IRC log on Sunday), unfortunately my job hasn't allowed it this week. I just wanted to do some thinking out loud about both methods (voting/pkgstats) for both packages already in community and those that might get there in the future from a regular user's point of view (also with regards to privacy/paranoia matters). (1) pkgstats The obvious problem with accuracy is that not everybody will use it (or use it even from time to time to update their "contribution" to the statistics). Some people don't know about it, some people won't be bothered, some might be concerned about privacy. Even though IP address is not necessarily an identifier of a person, it still a "good enough information". I actually more or less trust Arch devs that really only a hash of the IP is stored together with the package list but I hardly can be sure and there are much more paranoid users out there than myself. (Their problem doesn't have to be only with privacy itself - when someone knows the packages you use and even the exact versions, it makes it so much easier to target some kind of attack on the system.) On the other hand it can be nicely used to promote a package that is in unsupported. "Do you use this package? Do you want to see it in community? Have you run pkgstats on you system then?" It would be nice to see the statistics in AUR frontend, one could see how far the package is from the magic number that makes the package a good candidate for community (whatever the number will be). As for pruning of community as it is now (if it still is an issue, I'm not quite sure anymore). How about this. Pick a reasonable percentage (it doesn't have to be the same number as the one for new packages entering community, it can be lower) by whatever criteria (number of packages to prune, number of MB to save, ...), create a list of all the packages with usage below this number and create lists of these packages grouped by their maintainers. Then send the individual maintainer-lists to the maintainers with a note that they should consider whether or not these particular packages are really a good material for community. At the same time put the list of all those packages on the web, announce its existence in the latest news and tell people that if they see a package/packages they use and haven't yet run pkgstats, they should probably do it now, otherwise the package might be removed from community. Then wait for some time and look at the change in statistics (maybe there will be some, maybe there won't). (2) votes Again, not everybody uses it. Especially since voting means that you have to have an AUR account. Today everybody has tons of accounts at different internet services, ideally one should have as many passwords as possible, and people don't like to create yet another account (I know I don't). Frankly, if I hadn't needed those about 15 packages I now maintain in unsupported (because I hadn't found them there), I wouldn't have created an AUR account either. There's another problem with accuracy. Even users who have an account and vote don't vote for every single package they use. Especially many people (myself included) probably never voted for packages already in community. This makes the system usable for dealing with the transition unsupported -> community but not for the other way round. That, too, could be helped by similar approach as above - count packages with the least votes, create their list (lists) and urge people to vote for packages on this list if they use them a want to see them still in community in the future. The problem is that this way the privacy concerns will be even bigger. Right now if someone looked up which packages I voted for, it wouldn't give them much of an idea which packages I actually use (because I only voted for packages in unsupported and only for those that I had a reason to believe that my vote might help push them to community). After applying the above suggestion, anyone who gained access to AUR data knows more or less about all community packages that a certain nickname uses (which is much worse that knowing that this list of packages is used by someone with this hash of IP address - which is the information pkgstats provides). Moreover, each nickname is associated with an e-mail which is then more or less associated with a particular person. Of course, the e-mail can be fake (or completely or almost unused), on the other hand if you also want to maintain some packages in unsupported, you want to have a valid e-mail, so, if you're paranoid, you'd probably have to have two AUR accounts - one connected to you for maintaining packages and the other one as "anonymous" as possible just for voting. Conclusion Unfortunately, I don't have a solution. Both systems can be made more accurate (and useful for pointing fingers at packages that really aren't all that much used) but at the price of some amount of privacy or even security. I still think that the best solution would be counting downloads, because it would be quite accurate and also quite anonymous (definitely more than pkgstats or voting) but sadly it's not an option. I hope I haven't wasted too much time of those who have read it all. If so, then I apologize :-), but I felt that when I spent some the time thinking about these matters on my way to work and back this week, I should share the thoughts. Ondřej -- Cheers, Ondřej Kučera

On Wed, Dec 3, 2008 at 3:57 PM, Loui Chang wrote:

On Wed, Dec 03, 2008 at 10:15:23PM +0100, Kristoffer Fossgård wrote:

...

Why is package popularity judged by votes anyway? I never vote. The reason i never vote is because i don't understand why package popularity can't simply be judged by download count.

Votes are a conscious approval of a package. Downloads are not.

Interesting distinction I haven't heard spelled out in words: explicit approval vs implicit approval. 8)

kludge wrote:

Loui Chang wrote:

...

On Wed, Dec 03, 2008 at 10:15:23PM +0100, Kristoffer Fossgård wrote:

...

Why is package popularity judged by votes anyway? I never vote. The reason i never vote is because i don't understand why package popularity can't simply be judged by download count. Votes are a conscious approval of a package. Downloads are not.

explicit downloads sure are... unless you like to play pacman roulette, i guess.

but i don't. if it's explicitly installed, it's because i wanted the damn thing.

-kludge

sorry, that obviously only applies to [community] and to the aur. but it still applies to [community]. -kludge

On Wed, Dec 03, 2008 at 07:21:32PM -0600, kludge wrote:

Loui Chang wrote:

...

On Wed, Dec 03, 2008 at 10:15:23PM +0100, Kristoffer Fossgård wrote:

...

Why is package popularity judged by votes anyway? I never vote. The reason i never vote is because i don't understand why package popularity can't simply be judged by download count.

Votes are a conscious approval of a package. Downloads are not.

explicit downloads sure are... unless you like to play pacman roulette, i guess.

but i don't. if it's explicitly installed, it's because i wanted the damn thing.

That's good. I occaisionally download apps to try them and remove them soon after. Those are packages I would not vote on. Don't tell me you haven't ever done that.

On Thu, Dec 4, 2008 at 9:54 AM, Kristoffer Fossgård wrote:

Your all missing my point. I never said counting packages by downloadrate is a perfect solution but that IT IS GOOD ENOUGH _and_ BETTER THAN THE VOTE SYSTEM.

That's what I thought. Even monitoring a single download mirror could be enough, if it's not an obscure and unpopular one. At least gathered data would be statistically *relevant*, even though not accurate. We can think of a single mirror as a good approximation of the whole community, excluding i18n/l10n packages, which are highly dependendt on the physical location of the mirror itself. Corrado

Ronald van Haren wrote:

On 12/4/08, bardo wrote:

...

On Thu, Dec 4, 2008 at 9:54 AM, Kristoffer Fossgård wrote:

...

Your all missing my point. I never said counting packages by downloadrate is a perfect solution but that IT IS GOOD ENOUGH _and_ BETTER THAN THE VOTE SYSTEM.

That's what I thought. Even monitoring a single download mirror could be enough, if it's not an obscure and unpopular one. At least gathered data would be statistically *relevant*, even though not accurate. We can think of a single mirror as a good approximation of the whole community, excluding i18n/l10n packages, which are highly dependendt on the physical location of the mirror itself.

Corrado

I would think that the usage of packages depend on geographical location in the same way as distribution usage depend on geographical location.

ronald

How would any one mirror be any less biased that people submitting data via pkgstats? You are just sampling a different subset of people. Allan

On Thu, Dec 4, 2008 at 9:10 AM, Aaron Griffin wrote:

On Thu, Dec 4, 2008 at 5:52 AM, bardo wrote:

...

On Thu, Dec 4, 2008 at 9:54 AM, Kristoffer Fossgård wrote:

...

Your all missing my point. I never said counting packages by downloadrate is a perfect solution but that IT IS GOOD ENOUGH _and_ BETTER THAN THE VOTE SYSTEM.

That's what I thought. Even monitoring a single download mirror could be enough, if it's not an obscure and unpopular one. At least gathered data would be statistically *relevant*, even though not accurate. We can think of a single mirror as a good approximation of the whole community, excluding i18n/l10n packages, which are highly dependendt on the physical location of the mirror itself.

Guys. I have to point out a flaw in this reasoning. We are talking about packages _entering_ community. Not remaining there. For packages not in community, there is no download except from the AUR website. We *could* in theory, track this, but there's 3 or 4 different ways one can download things from the AUR

Again, just downloading a package does not mean I like it or use it. As someone previously stated: if you tell me you've never installed a packaged, tried it, and removed it because you didn't like it, you're probably lying.

I install packages all the time I then remove. And I am not lying either. Really I do !! Bob F.

On Thu, Dec 4, 2008 at 5:52 AM, bardo wrote:

...

On Thu, Dec 4, 2008 at 9:54 AM, Kristoffer Fossgård wrote:

...

Your all missing my point. I never said counting packages by downloadrate is a perfect solution but that IT IS GOOD ENOUGH _and_ BETTER THAN THE VOTE SYSTEM.

That's what I thought. Even monitoring a single download mirror could be enough, if it's not an obscure and unpopular one. At least gathered data would be statistically *relevant*, even though not accurate. We can think of a single mirror as a good approximation of the whole community, excluding i18n/l10n packages, which are highly dependendt on the physical location of the mirror itself.

Guys. I have to point out a flaw in this reasoning. We are talking about packages _entering_ community. Not remaining there. For packages not in community, there is no download except from the AUR website. We *could* in theory, track this, but there's 3 or 4 different ways one can download things from the AUR

There's one way technically. You download the tarball. Where are all the other ways? Even if there are why is this even relevant? It's not like a reasonably good-enough download counter is hard technically to accomplish(feel free to scold me if you think it is).

Again, just downloading a package does not mean I like it or use it. As someone previously stated: if you tell me you've never installed a packaged, tried it, and removed it because you didn't like it, you're probably lying.

Your still not getting it. The system doesn't have to be 100% perfect, it only has to offer a representation of which packages are "popular". that's it. we don't need to know how many "downloads" are really "conscientious" because the large majority of them will be. -K (ps: i'm sorry if i'm a bit harsh in this email but i haven't slept much today :))

There as been a lot of good discussion, and it appears there are more or less two "sides" here. Perhaps it would be a good idea for people to try to summarize the argument of the "opposing side", to see if the two groups really understand each other's positions. I've seen a bunch of good points made by proponents of either side, but there's a danger that they're being lost in the mailing list deluge. A concise list of of the pros and cons of the various courses of action might be a helpful tool, too -- editable by all on a wiki page, perhaps. Just an idea :). Drew On Thu, Dec 4, 2008 at 10:37 AM, Allan McRae wrote:

Kristoffer Fossgård wrote:

...

...

On Thu, Dec 4, 2008 at 5:52 AM, bardo wrote:

...

On Thu, Dec 4, 2008 at 9:54 AM, Kristoffer Fossgård wrote:

...

Your all missing my point. I never said counting packages by downloadrate is a perfect solution but that IT IS GOOD ENOUGH _and_ BETTER THAN THE VOTE SYSTEM.

That's what I thought. Even monitoring a single download mirror could be enough, if it's not an obscure and unpopular one. At least gathered data would be statistically *relevant*, even though not accurate. We can think of a single mirror as a good approximation of the whole community, excluding i18n/l10n packages, which are highly dependendt on the physical location of the mirror itself.

Guys. I have to point out a flaw in this reasoning. We are talking about packages _entering_ community. Not remaining there. For packages not in community, there is no download except from the AUR website. We *could* in theory, track this, but there's 3 or 4 different ways one can download things from the AUR

There's one way technically. You download the tarball. Where are all the other ways? Even if there are why is this even relevant? It's not like a reasonably good-enough download counter is hard technically to accomplish(feel free to scold me if you think it is).

...

Again, just downloading a package does not mean I like it or use it. As someone previously stated: if you tell me you've never installed a packaged, tried it, and removed it because you didn't like it, you're probably lying.

Your still not getting it. The system doesn't have to be 100% perfect, it only has to offer a representation of which packages are "popular". that's it. we don't need to know how many "downloads" are really "conscientious" because the large majority of them will be.

The two systems we already have "offer a representation of which packages are popular" but there is much debate about how good that representation is. A third is really not going to help....

Allan

On Thu, Dec 4, 2008 at 8:10 AM, Aaron Griffin wrote:

On Thu, Dec 4, 2008 at 5:52 AM, bardo wrote:

...

On Thu, Dec 4, 2008 at 9:54 AM, Kristoffer Fossgård wrote:

...

Your all missing my point. I never said counting packages by downloadrate is a perfect solution but that IT IS GOOD ENOUGH _and_ BETTER THAN THE VOTE SYSTEM.

That's what I thought. Even monitoring a single download mirror could be enough, if it's not an obscure and unpopular one. At least gathered data would be statistically *relevant*, even though not accurate. We can think of a single mirror as a good approximation of the whole community, excluding i18n/l10n packages, which are highly dependendt on the physical location of the mirror itself.

Guys. I have to point out a flaw in this reasoning. We are talking about packages _entering_ community. [...]

This is a good point. However, some people have mentioned that it would be nice to prune [community] of unused packages, too, if we had a reliable way of detecting them. Thus, I think it makes sense to consider tracking [community] package usage as well if a new usage tracking system is developed.

Again, just downloading a package does not mean I like it or use it. As someone previously stated: if you tell me you've never installed a packaged, tried it, and removed it because you didn't like it, you're probably lying.

I do this all the time as well. One possible solution: if an "I use this!" message were sent to the stat-tracking server automatically from pacman upon installing a package, it would not be much of an extension to send a "Oops, not anymore" message when it is uninstalled. Thoughts?

Firmicus wrote:

I basically agree that we need some form of control.

However I have some reservations and comments.

First, I would like to understand the usefulness of discussing and voting on this proposal first, before tackling the (IMO probably more serious issue) of having too many packages with very low popularity / usefulness in our community repo. Perhaps the former issue is easier to deal with? Or could serve as a basis for dealing with the latter one?

Now to the specific points of the proposal:

...

* Only "popular" packages may enter the repo, as defined by 1% usage from pkgstats or 10 votes on the AUR.

Allan, can you develop a bit more what is your criterion for the figures 1% or 10 votes. Why not 5% or 3 votes? (which in my humble opinion would seem to make more sense...)

...

* Any additions not covered by the above criteria must first be proposed on the aur-general mailing list, explaining the reason for the exemption (e.g. renamed package, new package) at which point a general consensus from the TUs will be reached. TUs with large numbers of "non-popular" packages are more likely to be rejected.

What should a "general consensus" look like? If 5 TUs are for and 3 are against and the remaining ones do not reply, do we say yes or no? Can we be more specific on the above point without burdening the procedure with bureaucratic rules?

...

Stefan Hussmann wrote:

...

Some thoughts. - If we encourage people to drop packages that are not popular, we should also encourage them to take packages in "usupported" that _are_ popular to "community".

That would be the idea.

I have examined the repos extra, community and unsupported quite carefully, and as a result I do not think there are that many packages in unsupported that really deserve being in community! Possible candidates are

yaourt, etc: but only if wain would become a TU AND if we think it is a good idea to support that very popular tool (not sure personally – also it is only a script, so easy to install). aurup nspluginwrapper-flash + dependencies > well, these are no longer required I believe splashy wine-doors songbird uswsusp customizepkg pactools archassistant vlc-plugin fbsplash

Otherwise, I would rather VERY MUCH encourage TUs to have a look at the packages in EXTRA that are currently orphaned, and to contact the devs if you are interested to maintain some of them in community. There are many more important packages there that would deserve to be maintained in community than being orphaned in extra.

That's it for now.

F

Little thing here: Never add 32bit binaries into 64bit system on community front (nspluginwrapper above), libs are kinda ok. Atleast I strongly disagree on adding them. - Neverth