[pacman-dev] [PATCH v2] Swap alpm_db_update() implementation to multiplexed version

[guillaume at manjaro.org]

Le 2020-05-11 17:10, Eli Schwartz a écrit :
> On 5/11/20 10:27 AM, guillaume at manjaro.org wrote:
>> Hi Allan, Hi Anatol,
>> 
>> I was looking to this new multiplexed implementation and I found some
>> problems:
>> 1) This implementation can lead to download databases from different
>> servers, I think it can't be a problem if they are not synced
> 
> This should already be the case in the single-download version too, no?
> 

I agree and nothing can be done except ensuring all databases are 
downloaded
from the same server.

>> 2) Differents download payloads are created for databases and its
>> signature files, this can this time lead to download the database and
>> its signature from differents server, I'm sure it's not wanted.
>> 3) Because signature files have their own download payloads, if the
>> signature is missing on a the first server (e.g. database signature is
>> optional), all next servers will be tried and all will give the same 
>> error.
> 
> re 2) I cannot remember offhand if this was already the case, but 
> anyway
> it's not impossible for one to download the database successfully from
> one mirror, then get a sudden network issue on the signature. Falling
> back to a second server for optional signatures will just lead to
> download errors (which are ignored for sigs), and for required
> signatures it will fail the signature check the same way failure to
> download the signature at all would fail, because one way or another we
> *need* an updated signature.
> 

I agree that the signature checking will fail anyway but we don't need 
to start
downloading a signature if we know it's from a different server than the 
one used for the database.

> re both 2) and 3) I guess this could be optimized in the database
> download handling to abort on the first 404 but fall through to
> additional servers for connection failures. I'm not sure whether this
> would be a good thing... how would we detect mirrors that can be
> connected to, but no longer host any pacman repos? Checking 5 mirrors
> for 404 errors isn't much of an issue.
> 
> I guess it could also solve this error I got:
> 
> error: failed retrieving file 'core.db.sig' from 
> mirror.epiphyte.network
> : Could not resolve host: mirror.epiphyte.network
> error: failed retrieving file 'testing.db.sig' from
> mirror.epiphyte.network : Could not resolve host: 
> mirror.epiphyte.network
> error: failed retrieving file 'extra.db.sig' from
> mirror.epiphyte.network : Could not resolve host: 
> mirror.epiphyte.network
> error: failed retrieving file 'community-testing.db.sig' from
> mirror.epiphyte.network : Could not resolve host: 
> mirror.epiphyte.network
> error: failed retrieving file 'community.db.sig' from
> mirror.epiphyte.network : Could not resolve host: 
> mirror.epiphyte.network
> error: failed retrieving file 'multilib.db.sig' from
> mirror.epiphyte.network : Could not resolve host: 
> mirror.epiphyte.network
> error: failed retrieving file 'multilib-testing.db.sig' from
> mirror.epiphyte.network : Could not resolve host: 
> mirror.epiphyte.network
> 
> But I'd already solved this one by removing the dead mirror from the
> bottom of my mirrorlist (it used to be a pretty good fallback one). 
> When
> all mirrors fail in the more normal manner at retrieving a signature,
> there's no UI results.
> 

Yes it should be optimized, I think we should simply not retry to 
download a
signature after a failure.
This is the point that makes me first inspect this code, it leads to try 
to download the
signature from every mirror for every database so it's can really slow 
down the process.

>> I really wanted to suggest a patch but I didn't find a easy way
>> (understand a small patch) to fix it and I would like your opinion on
>> point 1) first.
>> 
>> Regards,
>> Guillaume.