[arch-general] Making pacman check multiple repos

[Ng Oon-Ee]

On Sun, 2009-12-13 at 03:31 -0500, Qadri wrote:
> On Sun, Dec 13, 2009 at 2:07 AM, Brendan Long <korin43 at gmail.com> wrote:
> 
> > On Sun, 2009-12-13 at 11:16 +0800, Ng Oon-Ee wrote:
> > > On Sat, 2009-12-12 at 17:08 -0700, Brendan Long wrote:
> > > > 2009/12/11 Ng Oon-Ee <ngoonee at gmail.com>
> > > >
> > > > > On Sat, 2009-12-12 at 02:13 +0100, Heiko Baums wrote:
> > > > > > Am Sat, 12 Dec 2009 08:58:17 +0800
> > > > > > schrieb Ng Oon-Ee <ngoonee at gmail.com>:
> > > > > >
> > > > > > > Because sometimes all the mirrors listed in mirrorlist will not
> > have
> > > > > > > the file, if its just been uploaded. Also not everyone stays
> > > > > > > up-to-the-minute with updates, judging by the "updated after a
> > month"
> > > > > > > posts we see once in a while.
> > > > > > >
> > > > > > > I'm concerned about the last bit, if a package was just uploaded
> > and
> > > > > > > only exists on one mirror, everyone who updates and has that
> > package
> > > > > > > in the period between its uploading and its appearance on their
> > local
> > > > > > > mirrors will 'fall-back' on varying mirrors (lengthening the
> > update
> > > > > > > process) and all end up on the poor main server (or Tier 1/2
> > mirrors).
> > > > > > > Bad for both the mirror bandwidth as well as most probably much
> > slower
> > > > > > > for the user, who could probably just wait a day or so for the
> > update
> > > > > > > to come to his (faster, presumably) local mirror.
> > > > > >
> > > > > >
> > > > > > Wouldn't it be possible to first upload the packages and update the
> > db
> > > > > > files when the packages on the mirrors (at least on several
> > mirrors)
> > > > > > are updated?
> > > > > >
> > > > > > If I have such a "problem" that a package is on no mirrors, which
> > > > > > doesn't happen often, I usually abort the system update and wait
> > one
> > > > > > day. I think that's the normal and easiest way of solving this
> > issue.
> > > > > >
> > > > > > Greetings,
> > > > > > Heiko
> > > > >
> > > > > The few mirrors which sync first would have quite much higher
> > bandwidth
> > > > > usage =).
> > > > >
> > > > > The concern then is that in the period of time between uploading of
> > > > > packages and updating of db, the db would point to a package
> > (foo-1.3)
> > > > > while the mirror would only have the new version (foo-1.4), since I
> > > > > don't think many mirrors keep multiple copies of the same package
> > > > > (schlunix I know off, any others?). So that would break updating as
> > > > > well, just in a different direction, and this would not be
> > recoverable
> > > > > from.
> > > > >
> > > > >
> > > > Maybe a better idea would be to make pacman keep track of the last time
> > it
> > > > got an updated package list, and if it's beyond a certain point, it
> > starts
> > > > checking other mirrors (maybe optional "No updates have been found in 5
> > > > days, would you like to scan other mirrors for updates?").
> > >
> > > The difference I would see is that in the current system an out-of-date
> > > mirror is still use-able (no mismatch between db and package as we're
> > > discussing). Some would manually change mirrors, but others would not,
> > > so the net effect of automating fallover would be to increase load above
> > > what it currently is.
> > >
> > > Not everyone needs packages right on the day they're uploaded, or runs
> > > pacman everyday (I do, though =p)
> > >
> >
> > Not everyone runs pacman every day, but when you do, you expect to get
> > up to date packages.
> >
> >
> 
> So should it be a function of the program to make sure that happens? Or is a
> responsibility of the user? Should the functionality be programmed into
> pacman to make sure that happens, or should we be asking that users be aware
> of what repos they're using?

Well said, I agree. I believe that if separate db and package downloads
are implemented it should not be so users can be 'up-to-the-minute' in
packages, but for greater security.

In fact, now that I think about it, having two dbs (one on the mirror
with all packages as available on that mirror and one 'master' with a
list of authoritative checksums) would make sense, as it fulfils the
security aspect well while avoiding the problem of db/package mismatch.
The 'master' db would have to have a history of previous checksums as
well.