On Mon, Jan 26, 2009 at 4:17 PM, Grigorios Bouzakis grbzks@gmail.com wrote:

On Mon, Jan 26, 2009 at 02:39:38PM -0600, kludge wrote:

...

if there is no voting on packages in [community], then what mechanism exists for users to suggest/cheer on a package's promotion to [extra]?

-kludge

One users answer:

Maybe the same mechanism that would move packages out of extra to unsupported or community? Its called pkgstats. pacman -S pkgstats and then exec pkgstats as root IIRC.

Greg

Holy smokes, the current direction things are going in is scaring me. I don't understand why so much reliance has been put into pkgstats when there are a lot of fundamental problems with it in its current form. These are the EXACT same concerns that Bob Finch and I brought up with the recent vote on community package guidelines, and it seems everyone has been ignoring those issues and is ready to put all sorts of trust into pkgstats without examining its pitfalls.

For starters, pkgstats has been around only since the beginning of November of last year (a little less than three months now). Although its implementation is more simple by design, we did have ArchStats before that, and people had NOWHERE near as much faith in that tool's statistics as they apparently do in pkgstats. Yes ArchStats was more complicated (and unmaintained), but it also had other features and protections that pkgstats does not. I agree that simplicity is nice from a development standpoint, but I'd argue that it also has the potential to lower the tool's accuracy.

As far as implementation issues, right now, pkgstats does not accommodate for multiple machines with the same IP address (extremely common in home environments running NAT). It also cannot track changes over time or expire old data, which I'd think would be an absolute necessity if you want want to be able to rely on it for getting an accurate snapshot of current installation numbers or see to trends (which seems to be what people want the tool to be used for).

Which brings me to the next set of problems. How do you know the data is an accurate reflection of what the community wants or uses? Right now, people are using the terms "popularity", "usage", and "installed" as the same thing and assuming that is what pkgstats shows them. The script only reflects what is installed, not what people are using, not what people are interested in...just installed! What about people who download things to try them out, but do not remove them? What about packages that don't have old dependencies removed when they're updated? There are a bunch of scenario's where long-running systems might have unnecessary or stale packages installed that the user is unaware of (see liblbxutil at 67.74%, csup at 42.53%, and xorg-xsm at 34.74% as reported by pkgstats).

Also, pkgstats may be the best thing we currently have, but how do we know the stats it generates are an accurate representation of Arch users? Do we know how many Arch users there are in total? We have just over 20,000 registered forum members (the US forum only), just under 13,000 registered AUR users, and just about 3000 unique IPs that have contributed to pkgstats. The thing is, who knows how many Arch users are out there that haven't registered for any of our sites? Or what about users who have registered for our sites but no longer use Arch? Or users with multiple machines and the issue of whether or not those machines should count differently for pkgstats? Who knows how many of those unique IP addresses are unique machines and not just updates from the same machine where an ISP has handed out a different IP address? If there are users out there that are unaware of pkgstats, or aren't that involved in the community (but still use Arch), then are the current reports skewed one way or another? Meaning, are the packages that developers and people active in the community reflect the packages used by everyone?

Also, when pkgstats was introduced, it was said that "In an ideal world one would run the script only once per installation or if really lot of things have changed (not the version of packages).", but if you want these numbers to be up-to-date, then it SHOULD be run on a regular basis and the old data should be discarded or weighted differently. Right now, there have been 3,757 pkgstat submissions, meaning about 700 people have run it multiple times. If we want people to keep up that habit and maintain current info, then perhaps it should come with a script that could be enabled to run it as a cron job...even if it does not get enable by default.

Not only that, but pkgstats reports that no package has a 100% installation percentage...even pkgstats itself only has 98.88%, which means people have stuff installed on their systems that have not been reported. In this particular case, it's probably people that have downloaded the script directly rather than installed it via pacman, but it's still an issue to consider with other packages as well.

Because pkgstats is not an "official" tool that is distributed with the core and turned on by default (which I don't think it should be), that alone means it has some amount of bias built into it. Not knowing the answers to a lot of these questions means that it's incredibly hard to tell statistically if the numbers reported by pkgstats are accurate or should be relied upon, and yet that's what people are doing.

My whole point in bringing all this up is that it seems like people are treating pkgstats as a panacea, when I don't think it should be the sole basis of all of these recent decisions. I absolutely support the idea of pkgstats, but think that its still in its infancy and should not be so blindly trusted. People have been suggesting arbitrary limits and restrictions based on its numbers, but the numbers themselves could be pretty far off base...and I think that's the bigger problem.

-- Aaron "ElasticDog" Schaefer

Loui Chang wrote: <snip />

There's no point in writing novels if it's just fantasy.

unless you're tolkien, eddings, leguin, etc., etc. :b

On Mon, Jan 26, 2009 at 7:30 PM, Aaron Schaefer aaron@elasticdog.com wrote:

On Mon, Jan 26, 2009 at 8:13 PM, Loui Chang louipc.ist@gmail.com wrote:

...

There's no point in writing novels if it's just fantasy.

I acknowledge that I don't have solutions for all of the issues I bring up, but that doesn't mean the issues aren't important, nor should fixing them be considered "fantasy". I just don't understand why these things are being brushed off and ignored by nearly everyone...ignorance is bliss?

Serious note... we were talking about implementation details of this switch, were we not? You guys can keep talking about the relative merits of different statistical gathering procedures, but it makes me not care about doing this work at all... can we please stick to the topic so that this will actually get done? If not, feel free to send me patches.

On Mon, Jan 26, 2009 at 11:40 PM, Allan McRae allan@archlinux.org wrote:

So your point is what exactly and how does it relate to the topic at hand?

Allan

It relates because of kludge's question of "what mechanism exists for users to suggest/cheer on a package's promotion to [extra]?" and Greg's response of "Its called pkgstats." and Loui's suggestion of doing a cleanup based on pkgstats/votes before this transition:

On Mon, Jan 26, 2009 at 2:26 PM, Loui Chang louipc.ist@gmail.com wrote:

It seems the consensus is to remove community packages from the AUR interface altogether.

Before we do that though we should do a general package clean up. Moving packages with extremely low votes/pkgstats usage into unsupported, so we're not importing garbage into the new system.

What do you think?

You're asking for people's suggestions on how to proceed with this transition, and I'm merely pointing out that some of the proposed implementation details might not be in the best interest for Arch users...or more accurately, there's no way to tell if they are or not. Yet, we keep pushing forward to change things regardless and basing decisions on the same false pretenses.

I think kludge's question is completely valid, and combining these tools does blur the line on management policies and how extra/community have worked in the past. Dismissing that concern by saying there's nothing we can do about losing votes and we'll just rely on pkgstats might not be the best solution. So my concerns about pkgstats (and votes) may not be an implementation detail, per se, but it is absolutely related.

-- Aaron "ElasticDog" Schaefer

On Tue, Jan 27, 2009 at 12:54 AM, Aaron Griffin aaronmgriffin@gmail.com wrote:

Please please please for the sake of our sanity - just stop. . . . I really don't like the "it's broken so don't change anything" attitude. This is about packages and TUs. Not about votes - I have no idea why we're talking about this.

I'm trying to share my thoughts on the matter because I do think they're relevant, and I care about Arch's development...I'm not here to argue just for argument's sake. So if you want me to sit back and shut up, I will, but It seems like a lot of people have selective reading, and many conversations on the list are completely one-sided. I don't speak up often, but when I do it's usually my goal to provide a little balance where there is none. I have no secret agenda. If you think eliminating votes as a side effect of this transition, or performing a cleanup based on pkgstats prior to integrating the database has nothing to do with this process, then so be it...

I agree that the "it's broken so don't change anything" attitude is not constructive (nor is that my attitude), but an attitude of "it's broken, so let's make changes just because it will be different" is no better if those changes aren't an improvement.

-- Aaron "ElasticDog" Schaefer

On Mon, Jan 26, 2009 at 10:24 PM, Aaron Schaefer aaron@elasticdog.com wrote:

I agree that the "it's broken so don't change anything" attitude is not constructive (nor is that my attitude), but an attitude of "it's broken, so let's make changes just because it will be different" is no better if those changes aren't an improvement.

also. Don't kick a running horse. I swear that statement is somehow relevant.