Well, nowadays I started to dislike requiredby: 1. The requiredby handling was _very_ negligent earlier, so most users have corrupt localdbs. 2. When the old deps.c just trusted on these fields blindly (-R), these db corruptions caused wrong pacman operation too. After we implemented a slower but more clever checkdeps, all of these problems disappeared. 3. So now a missing requiredby can generate a bug, a false cannot. Note: we use requiredby as a speed-up only (probably speed-up was the main reason for implementing it). 4. Do we really need this speed-up? a.) This is not as efficient as it could be: %REQUIREDBY% should show the dependency also [%REQUIREDBY% gcc:bash>=3.2]; since now foo's %REQUIREDBY% says gcc only, so we have to _scan_ pkgcache for gcc; then check _all_ dependencies of gcc to find the foo-satisfies-this ones. b.) If we didn't use requiredby at all, we should just scan pkgcache (only once!, now we scans it n times, if foo has n requiredby fields), and simply check all depends of all packages to find the satisfied-by-foo dependencies. This would probably slower, and evaluate its degree, but testdb is extremely fast, which makes me optimist. What's more, this is not a speed-up at all! So I'm pretty sure now, that storing requiredby is _useless_ (no, even worse: a bug generator): I simply forgot about update_requiredby! Requiredby is computed when the package was born on localdb <- requiredby computation is ~exactly 4./b.). This result is used only in the following cases: *) by pacman: -Si and -Qi (wow), --orphan *) deps.c/checkdeps && can_remove *) libalpm/sync.c:566,1256 what are these?! we don't copy requiredby fields on package upgrade neither, why should we copy it between to-be-replaced and replacer?! [And I didn't say anything about trans_update_depends, grr.] Summary: if we just simply followed 4./b.), we would compute requiredby on package death instead of on package born. The question is: how many times we use this information without removal (-Qi, -Si and can_remove, --orphan). But as testdb shows: compute requiredby for _all_ packages is not costly at all.

So I will create a patch for killing requiredby soon. The patched pacman will be compatible with the old dbs; however, old pacmans with new dbs will fail. What do you say? Any contras? [my patch would simulate alpm_pkg_get_requiredby with compute_requiredby] Bye, ngaba ---------------------------------------------------- SZTE Egyetemi Könyvtár - http://www.bibl.u-szeged.hu This mail sent through IMP: http://horde.org/imp/

On Nov 12, 2007 10:21 AM, Nagy Gabor <ngaba@bibl.u-szeged.hu> wrote:

So I will create a patch for killing requiredby soon. The patched pacman will be compatible with the old dbs; however, old pacmans with new dbs will fail. What do you say? Any contras? [my patch would simulate alpm_pkg_get_requiredby with compute_requiredby]

Just to make sure I understand you - you want to fully remove REQUIREDBY from the DB and compute it every time. It will be a little slower, BUT, it will help us in a few ways: 1) we don't need REQUIREDBY all that often. It's only used in removal, and replace functionality if I understand correctly. The performance hit will not be that big in this case (removal happens far less often than addition). 2) Our package structures will shrink in size, which is always good I say go for it. Sounds like a pretty decent idea if I'm understanding you correctly. As usual, I await your patches 8)

On Nov 12, 2007 2:18 PM, Travis Willard <travis@archlinux.org> wrote:

On Mon, 12 Nov 2007 14:08:07 -0600, Aaron Griffin wrote

...

On Nov 12, 2007 10:21 AM, Nagy Gabor <ngaba@bibl.u-szeged.hu> wrote:

...

So I will create a patch for killing requiredby soon. The patched pacman will be compatible with the old dbs; however, old pacmans with new dbs will fail. What do you say? Any contras? [my patch would simulate alpm_pkg_get_requiredby with compute_requiredby]

Just to make sure I understand you - you want to fully remove REQUIREDBY from the DB and compute it every time.

It will be a little slower, BUT, it will help us in a few ways:

1) we don't need REQUIREDBY all that often. It's only used in removal, and replace functionality if I understand correctly. The performance hit will not be that big in this case (removal happens far less often than addition). 2) Our package structures will shrink in size, which is always good

3) Computing requiredby every time will ensure (if the computation is done right) that requiredby will always be correct. Even after -Rd operations followed by subsequent -S operations and other crazy things. Basically, less bookkeeping to do on install/remove.

I knew I had more points in my head! This was one of them - preventing DB corruption. 4) Less disk writes, as we actually have to write, compute, and rewrite the entry, if memory serves me correctly.

On Nov 12, 2007 3:35 PM, Dan McGee <dpmcgee@gmail.com> wrote:

On Nov 12, 2007 2:28 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:

...

On Nov 12, 2007 2:18 PM, Travis Willard <travis@archlinux.org> wrote:

...

On Mon, 12 Nov 2007 14:08:07 -0600, Aaron Griffin wrote

...

On Nov 12, 2007 10:21 AM, Nagy Gabor <ngaba@bibl.u-szeged.hu> wrote:

...

So I will create a patch for killing requiredby soon. The patched pacman will be compatible with the old dbs; however, old pacmans with new dbs will fail. What do you say? Any contras? [my patch would simulate alpm_pkg_get_requiredby with compute_requiredby]

Just to make sure I understand you - you want to fully remove REQUIREDBY from the DB and compute it every time.

It will be a little slower, BUT, it will help us in a few ways:

1) we don't need REQUIREDBY all that often. It's only used in removal, and replace functionality if I understand correctly. The performance hit will not be that big in this case (removal happens far less often than addition). 2) Our package structures will shrink in size, which is always good

3) Computing requiredby every time will ensure (if the computation is done right) that requiredby will always be correct. Even after -Rd operations followed by subsequent -S operations and other crazy things. Basically, less bookkeeping to do on install/remove.

I knew I had more points in my head! This was one of them - preventing DB corruption.

4) Less disk writes, as we actually have to write, compute, and rewrite the entry, if memory serves me correctly.

+1. I think we can trade a possible slight performance loss in a few cases for all the things mentioned above. I also think it will cut out a lot of crappy code.

Throwing the project manager speak into this here- it isn't getting into 3.1. This will be a great kickoff for 3.2 major changes though.

Along with real transactions, parsing cleanup, DB backend changes, a million dollars, naked women, and copious amounts of booze, yes?

> I'm regretting Nagy didn't point this out earlier then.
> I don't know why I never thought about this. Now that it's said, and I'm
> looking back at all the issues related to requiredby, it makes a lot of
> sense.
> It would also eliminate the eventual problems caused by an existing broken
> database, with wrong requiredby.
Well, I'm quoting Aaron here
(http://www.archlinux.org/pipermail/pacman-dev/2007-October/009498.html):
"The more and more ideas that are pumped out to this list, the more
confused we get, and the harder it is to keep up. It's like.... if I
throw a baseball at you, you might catch it, but if I throw 50, you
won't catch them all."
And he has right now. I've got plenty of plans in my mind, indeed; I didn't
point this out earlier, because:
1. Main reason: I always prefer bugfixes over "improvements" (see also: 2.). And
well, in my priority list this is not so important at all:
-give a look at libalpm/sync.c (simply horrible)
-we have some critical bugs now (symlink puzzle), and some other notable failing
pactests too, buggy reason handling etc.
-my main "mission" here is to reach the universal transaction ;-)
2. Now the requiredby stuff is expected to work fine (I can bet that Travis's
problem is not a bug now).
3. Throwing requiredby to trash-bin is also throwing our hard work to release
2., too
4. To be honest, I simply thought that you won't like it
5. I have some pending submitted patches here (I don't want to throw 50
baseballs... ;-)

> 
> But I'm not the project manager, and pacman 3.1 indeed has to be released
> one
> day :)
> 
Well, sometimes I have the feeling that we should release pacman more often. I
said earlier that pacman-git is more stable than devel repo. And indeed, I
simply use pacman-git at home, fortunately without bigger problems. Sometimes I
get disappointed about the fact, that Arch users have to wait so much for the
next release:
Pacman 3.0.6 fails 29 (vs. 8) pactest files from 3.1 (remove046.py lead to an
infinite loop...)

Bye, ngaba


----------------------------------------------------
SZTE Egyetemi Könyvtár - http://www.bibl.u-szeged.hu
This mail sent through IMP: http://horde.org/imp/

On Nov 14, 2007 5:01 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:

On Nov 12, 2007 3:35 PM, Dan McGee <dpmcgee@gmail.com> wrote:

...

Throwing the project manager speak into this here- it isn't getting into 3.1. This will be a great kickoff for 3.2 major changes though.

You sure you still want to stand by this? I think this change might actually close out like 3 or 4 bugs on the 3.1 tracker... I say push it, but I'll let you be the judge.

The project manager didn't realize how easy this change actually would be. :) If no one else has any objections, then I'd like to merge my changes tonight. We (anyone want to volunteer) also need to make a tool to remove requiredby entries. I took an initial stab at it using libalpm, then I realized we have no explicit mechanism to tell the backend to write to the DB. Does this sound like something we should expose, or is it too low level? BTW, the reason for the tool is simple (responding to earlier emails in the thread)- keeping stale entries around is confusing for users AND any scripts that may have used it. -Dan

On Wed, 2007-11-14 at 18:15 -0600, Aaron Griffin wrote:

On Nov 14, 2007 6:08 PM, Dan McGee <dpmcgee@gmail.com> wrote:

...

We (anyone want to volunteer) also need to make a tool to remove requiredby entries.

I was going to python it up... it's an easy file to parse and rewrite, probably like 20 lines

sed rules! :) grep -Rl REQUIREDBY /var/lib/pacman/local | xargs sed -i '/^%REQUIREDBY %/,/^$/ d' Course it may not handle corner cases... k -- K. Piche <kpiche@rogers.com>

On Nov 14, 2007 6:08 PM, Dan McGee <dpmcgee@gmail.com> wrote:

...

We (anyone want to volunteer) also need to make a tool to remove requiredby entries.

I was going to python it up... it's an easy file to parse and rewrite, probably like 20 lines

...

I took an initial stab at it using libalpm, then I realized we have no explicit mechanism to tell the backend to write to the DB. Does this sound like something we should expose, or is it too low level?

I'd say no. There's a reason why we don't have mutators for package structures (front ends shouldn't be modifying that data). If you implement this with care (don't let the front-end corrupt the db), this is acceptable for me. Personally, sometimes I modify my localdb by hand (mostly %REASON%), which is much more dangerous than a well controlled localdb->pmpkg_t...edit...pmpkg_t->localdb process. After we removed %REQUIREDBY%, %REASON% is the only reason in my mind where this can be useful [so I revert this: http://www.archlinux.org/pipermail/pacman-dev/2007-November/010066.html] See also: http://www.archlinux.org/pipermail/pacman-dev/2007-September/009302.html I know your answer: reinstall the package... This is needless, and now the REASON handling is chaotic, probably I must do -Rd, -S if I want to set a dependency reason to explicit [<- so as a brother of --asdep, --asexplicit can be useful too], which is ugly. Bye, ngaba

---------------------------------------------------- SZTE Egyetemi Könyvtár - http://www.bibl.u-szeged.hu This mail sent through IMP: http://horde.org/imp/

On Nov 15, 2007 10:42 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:

On Nov 15, 2007 4:39 AM, Nagy Gabor <ngaba@bibl.u-szeged.hu> wrote:

...

...

On Nov 14, 2007 6:08 PM, Dan McGee <dpmcgee@gmail.com> wrote:

...

We (anyone want to volunteer) also need to make a tool to remove requiredby entries.

I was going to python it up... it's an easy file to parse and rewrite, probably like 20 lines

...

I took an initial stab at it using libalpm, then I realized we have no explicit mechanism to tell the backend to write to the DB. Does this sound like something we should expose, or is it too low level?

I'd say no. There's a reason why we don't have mutators for package structures (front ends shouldn't be modifying that data). If you implement this with care (don't let the front-end corrupt the db), this is acceptable for me. Personally, sometimes I modify my localdb by hand (mostly %REASON%), which is much more dangerous than a well controlled localdb->pmpkg_t...edit...pmpkg_t->localdb process. After we removed %REQUIREDBY%, %REASON% is the only reason in my mind where this

...snip...

This is totally different from a design perspective. Dan mentioned a generic mechanism to write, where as you're suggesting a unique operation. A generic, public interface to modify and write to the DB is, in my opinion, a bad idea, but operations which do things (and write) within libalpm's control is fine

I actually wasn't referring to mutators to make that clear. I was thinking along the lines of a public alpm_pkg_force_write(pmpkg_t *pkg) or something. However, that still doesn't seem that clean, but I don't know. -Dan

On Wed, Nov 14, 2007 at 05:01:21PM -0600, Aaron Griffin wrote:

On Nov 12, 2007 3:35 PM, Dan McGee <dpmcgee@gmail.com> wrote:

...

Throwing the project manager speak into this here- it isn't getting into 3.1. This will be a great kickoff for 3.2 major changes though.

You sure you still want to stand by this? I think this change might actually close out like 3 or 4 bugs on the 3.1 tracker... I say push it, but I'll let you be the judge.

AFAIK, all these bugs were already fixed. That's also what Nagy pointed out in an earlier mail. http://www.archlinux.org/pipermail/pacman-dev/2007-November/010024.html "2. Now the requiredby stuff is expected to work fine (I can bet that Travis's problem is not a bug now). 3. Throwing requiredby to trash-bin is also throwing our hard work to release 2., too"

> On Nov 12, 2007 10:21 AM, Nagy Gabor <ngaba@bibl.u-szeged.hu> wrote:
> > So I will create a patch for killing requiredby soon. The patched pacman
> will be
> > compatible with the old dbs; however, old pacmans with new dbs will fail.
> > What do you say? Any contras? [my patch would simulate
> alpm_pkg_get_requiredby
> > with compute_requiredby]
> 
> Just to make sure I understand you - you want to fully remove
> REQUIREDBY from the DB and compute it every time.
Exactly.

> It will be a little slower, BUT, it will help us in a few ways:
> 
> 1) we don't need REQUIREDBY all that often. It's only used in removal,
> and replace functionality if I understand correctly. The performance
> hit will not be that big in this case (removal happens far less often
> than addition).
Note: upgrade is also a "replace"; so we must compute requiredby in this case,
too. But this is also done now; just not in the removal part: this is done in
the add part (now every package addition also induces a compute_requiredby... +
alpm_trans_update_depends, so even a bit more).
So requiredby is _computed_ now on foo package born (and some not notable
modifications happens if new "requiredby" packages are installed). So we waste
time for computing requiredby, iff the package won't be removed (depcheck error,
-Si, -Qi, --orphan), because in the package "life" we do this computation more
than once (on removal, we _will_ compute).
As I see, the effects in speed will be (we are talking about just seconds here):
-add_prepare won't change, add_commit will be faster
-remove_prepare will be slower, remove_commit won't change, so user must wait
longer for checkdeps errors and confirmation questions (negative impact) 
-so -A will be faster, -R will be slower than now
- -U/-S: prepare will be slower (effect of ~remove_prepare), commit will be
faster (effect of add_commit), so user must wait longer for checkdeps errors and
confirmation questions (negative impact); but overall it will be faster (because
usually more package-adding than package-removal happens here)

> 2) Our package structures will shrink in size, which is always good
> 
> I say go for it. Sounds like a pretty decent idea if I'm understanding
> you correctly. As usual, I await your patches 8)
OK.
Bye, ngaba


----------------------------------------------------
SZTE Egyetemi Könyvtár - http://www.bibl.u-szeged.hu
This mail sent through IMP: http://horde.org/imp/