[arch-general] Question about repository updates and atomicity
I'm trying to understand what may and may not go wrong between package uploads to a repository, repository synchronization via rsync (which seems to be the preferred method for mirrors to get their stuff) and downloads via pacman. Assume a developer uploads a new version of two packages to a repository. My understanding is that uploads delete the older version of the uploaded package currently in the repository. (unlike other package management systems e.g. debian's where multiple versions of the same package can reside in the same repository). An hour later, a mirror repository starts rsycing. While the rsync is ongoing, a user uses pacman to download both packages from the mirror. Question: what mechanisms exists, if any, to avoid that the user gets the old version of package A and the new version of package B? (assume the rsync is still ongoing while the download occurs) It seems the "safe" way of doing the rsync would be to take down the mirror during the critical section (when all old versions are removed and the new ones put there instead), and restart it afterwards? Or is there something somewhere that I'm not seeing? The debian guys probably keep the old version(s) around for exactly that reason... Any insight very much appreciated ... Cheers, Johannes.
On 15/08/13 17:08, Johannes Ernst wrote:
I'm trying to understand what may and may not go wrong between package uploads to a repository, repository synchronization via rsync (which seems to be the preferred method for mirrors to get their stuff) and downloads via pacman.
Assume a developer uploads a new version of two packages to a repository. My understanding is that uploads delete the older version of the uploaded package currently in the repository. (unlike other package management systems e.g. debian's where multiple versions of the same package can reside in the same repository).
An hour later, a mirror repository starts rsycing. While the rsync is ongoing, a user uses pacman to download both packages from the mirror.
Question: what mechanisms exists, if any, to avoid that the user gets the old version of package A and the new version of package B? (assume the rsync is still ongoing while the download occurs)
The database file will either have both new packages or no new packages.
It seems the "safe" way of doing the rsync would be to take down the mirror during the critical section (when all old versions are removed and the new ones put there instead), and restart it afterwards? Or is there something somewhere that I'm not seeing? The debian guys probably keep the old version(s) around for exactly that reason...
Any insight very much appreciated ...
On Aug 15, 2013, at 2:05, Allan McRae <allan@archlinux.org> wrote:
Question: what mechanisms exists, if any, to avoid that the user gets the old version of package A and the new version of package B? (assume the rsync is still ongoing while the download occurs)
The database file will either have both new packages or no new packages.
That's what I was hoping, but I'd love to know how that is implemented? Thanks, Johannes.
On Thu, Aug 15, 2013 at 6:34 PM, Johannes Ernst <jernst@cldstr.com> wrote:
That's what I was hoping, but I'd love to know how that is implemented?
Server-side, the developer stages several packages and then triggers a db-update. A new database is built and the old database file is replaced.
On Aug 17, 2013, at 0:06, Jan Alexander Steffens <jan.steffens@gmail.com> wrote:
Server-side, the developer stages several packages and then triggers a db-update.
A new database is built and the old database file is replaced.
What happens if that occurs during an rsync? Or is that prevented from happening? Similarly, what if a client updates from a mirror while the database file is being replaced by rsync? Thanks, Johannes.
On Sun, Aug 18, 2013 at 8:32 PM, Johannes Ernst <jernst@cldstr.com> wrote:
What happens if that occurs during an rsync? Or is that prevented from happening?
Similarly, what if a client updates from a mirror while the database file is being replaced by rsync?
You can't get a partially updated database, as the replacement of a database is atomic. Both our dbscripts and rsync ensure that, by using the basic rename-to-replace approach (see `man 2 rename`). However, this only applies to single databases. Before really big changes that affect multiple databases I stop rsync and wait for all connections to close, to ensure atomicity. For example, I do this when I move a major GNOME update between repositories, as that can take a couple of minutes. I can't speak for others, though.
participants (3)
-
Allan McRae
-
Jan Alexander Steffens
-
Johannes Ernst