Re: [pacman-dev] [PATCH v2] Swap alpm_db_update() implementation to multiplexed version
Hi Allan, Hi Anatol, I was looking to this new multiplexed implementation and I found some problems: 1) This implementation can lead to download databases from different servers, I think it can't be a problem if they are not synced 2) Differents download payloads are created for databases and its signature files, this can this time lead to download the database and its signature from differents server, I'm sure it's not wanted. 3) Because signature files have their own download payloads, if the signature is missing on a the first server (e.g. database signature is optional), all next servers will be tried and all will give the same error. I really wanted to suggest a patch but I didn't find a easy way (understand a small patch) to fix it and I would like your opinion on point 1) first. Regards, Guillaume.
On 5/11/20 10:27 AM, guillaume@manjaro.org wrote:
Hi Allan, Hi Anatol,
I was looking to this new multiplexed implementation and I found some problems: 1) This implementation can lead to download databases from different servers, I think it can't be a problem if they are not synced
This should already be the case in the single-download version too, no?
2) Differents download payloads are created for databases and its signature files, this can this time lead to download the database and its signature from differents server, I'm sure it's not wanted. 3) Because signature files have their own download payloads, if the signature is missing on a the first server (e.g. database signature is optional), all next servers will be tried and all will give the same error.
re 2) I cannot remember offhand if this was already the case, but anyway it's not impossible for one to download the database successfully from one mirror, then get a sudden network issue on the signature. Falling back to a second server for optional signatures will just lead to download errors (which are ignored for sigs), and for required signatures it will fail the signature check the same way failure to download the signature at all would fail, because one way or another we *need* an updated signature. re both 2) and 3) I guess this could be optimized in the database download handling to abort on the first 404 but fall through to additional servers for connection failures. I'm not sure whether this would be a good thing... how would we detect mirrors that can be connected to, but no longer host any pacman repos? Checking 5 mirrors for 404 errors isn't much of an issue. I guess it could also solve this error I got: error: failed retrieving file 'core.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'testing.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'extra.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'community-testing.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'community.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'multilib.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'multilib-testing.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network But I'd already solved this one by removing the dead mirror from the bottom of my mirrorlist (it used to be a pretty good fallback one). When all mirrors fail in the more normal manner at retrieving a signature, there's no UI results.
I really wanted to suggest a patch but I didn't find a easy way (understand a small patch) to fix it and I would like your opinion on point 1) first.
Regards, Guillaume.
-- Eli Schwartz Bug Wrangler and Trusted User
Le 2020-05-11 17:10, Eli Schwartz a écrit :
On 5/11/20 10:27 AM, guillaume@manjaro.org wrote:
Hi Allan, Hi Anatol,
I was looking to this new multiplexed implementation and I found some problems: 1) This implementation can lead to download databases from different servers, I think it can't be a problem if they are not synced
This should already be the case in the single-download version too, no?
I agree and nothing can be done except ensuring all databases are downloaded from the same server.
2) Differents download payloads are created for databases and its signature files, this can this time lead to download the database and its signature from differents server, I'm sure it's not wanted. 3) Because signature files have their own download payloads, if the signature is missing on a the first server (e.g. database signature is optional), all next servers will be tried and all will give the same error.
re 2) I cannot remember offhand if this was already the case, but anyway it's not impossible for one to download the database successfully from one mirror, then get a sudden network issue on the signature. Falling back to a second server for optional signatures will just lead to download errors (which are ignored for sigs), and for required signatures it will fail the signature check the same way failure to download the signature at all would fail, because one way or another we *need* an updated signature.
I agree that the signature checking will fail anyway but we don't need to start downloading a signature if we know it's from a different server than the one used for the database.
re both 2) and 3) I guess this could be optimized in the database download handling to abort on the first 404 but fall through to additional servers for connection failures. I'm not sure whether this would be a good thing... how would we detect mirrors that can be connected to, but no longer host any pacman repos? Checking 5 mirrors for 404 errors isn't much of an issue.
I guess it could also solve this error I got:
error: failed retrieving file 'core.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'testing.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'extra.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'community-testing.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'community.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'multilib.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network error: failed retrieving file 'multilib-testing.db.sig' from mirror.epiphyte.network : Could not resolve host: mirror.epiphyte.network
But I'd already solved this one by removing the dead mirror from the bottom of my mirrorlist (it used to be a pretty good fallback one). When all mirrors fail in the more normal manner at retrieving a signature, there's no UI results.
Yes it should be optimized, I think we should simply not retry to download a signature after a failure. This is the point that makes me first inspect this code, it leads to try to download the signature from every mirror for every database so it's can really slow down the process.
I really wanted to suggest a patch but I didn't find a easy way (understand a small patch) to fix it and I would like your opinion on point 1) first.
Regards, Guillaume.
On 12/5/20 3:14 am, guillaume@manjaro.org wrote:
Le 2020-05-11 17:10, Eli Schwartz a écrit :
On 5/11/20 10:27 AM, guillaume@manjaro.org wrote:
Hi Allan, Hi Anatol,
I was looking to this new multiplexed implementation and I found some problems: 1) This implementation can lead to download databases from different servers, I think it can't be a problem if they are not synced
This should already be the case in the single-download version too, no?
I agree and nothing can be done except ensuring all databases are downloaded from the same server.
Except... those database for repos that are hosted on different servers. Because not all repos are provided by the distro. As this was the case in single download code, I don't consider this a bug in the new code. Allan
Hello Guillaume On Mon, May 11, 2020 at 7:51 AM <guillaume@manjaro.org> wrote:
Hi Allan, Hi Anatol,
I was looking to this new multiplexed implementation and I found some problems: 1) This implementation can lead to download databases from different servers, I think it can't be a problem if they are not synced
Database files are downloaded independently. There is no any grouping or tracking where did each file come from. It is true both for current (multiplexed) and for the previous (sequential) downloader implementation. Thus there was and is always a possibility that databases can come from different mirrors e.g. in case of a flaky server.
2) Differents download payloads are created for databases and its signature files, this can this time lead to download the database and its signature from differents server, I'm sure it's not wanted. 3) Because signature files have their own download payloads, if the signature is missing on a the first server (e.g. database signature is optional), all next servers will be tried and all will give the same error.
I really wanted to suggest a patch but I didn't find a easy way (understand a small patch) to fix it and I would like your opinion on point 1) first.
Regards, Guillaume.
On 12/5/20 12:27 am, guillaume@manjaro.org wrote:
Hi Allan, Hi Anatol,
I was looking to this new multiplexed implementation and I found some problems: 1) This implementation can lead to download databases from different servers, I think it can't be a problem if they are not synced
As I said elsewhere, this could happen in the single download code too. So not a new issue. This could be fixed (but note not all repos have the same server - only those provided by a distro), but it is beyond this patch.
2) Differents download payloads are created for databases and its signature files, this can this time lead to download the database and its signature from differents server, I'm sure it's not wanted.
Signature files coming from different servers is fine. You probably have that all the time with source code downloaded from places with various load balancing. Signature files for servers in different sync states is an issue - only for databases, packages are fine.
3) Because signature files have their own download payloads, if the signature is missing on a the first server (e.g. database signature is optional), all next servers will be tried and all will give the same error.
This is an issue. And I guess the fix would fix #2 too. It would be good to open a bug for this.
I really wanted to suggest a patch but I didn't find a easy way (understand a small patch) to fix it and I would like your opinion on point 1) first.
Regards, Guillaume. .
participants (4)
-
Allan McRae
-
Anatol Pomozov
-
Eli Schwartz
-
guillaume＠manjaro.org