[pacman-dev] Incremental package index support?
Hi, I am an update junkie. I do "pacman -Suy" several times a day. One thing I do not like with current pacman is that it redownloads whole index file each time I update system. :: Synchronizing package databases... core 104.4 KiB 104K/s 00:01 [######################] 100% extra 1416.9 KiB 171K/s 00:08 [######################] 100% community 1916.6 KiB 204K/s 00:09 [######################] 100% It is 3.5M it total even if only just a few new packages were added. Multiply 3.5M to number of users and you'll get a huge number. IMHO it is waste of bandwidth and user time. From other side index is incremental by it nature. Could pacman use index format that fits incremental updates better? A naive proposal is to use compressed format for "full fetch" (e.g. for users who did not update more than a week), and uncompressed append-only index for incremental. On incremental update client sends request to server "give me append file file starting from position XXX till the end of file", so only delta from the last update will be fetched. Then pacman applies the delta to client index. What do you think about this idea?
Hi On Sun, Mar 24, 2013 at 10:57 AM, Anatol Pomozov <anatol.pomozov@gmail.com> wrote:
Hi,
I am an update junkie. I do "pacman -Suy" several times a day. One thing I do not like with current pacman is that it redownloads whole index file each time I update system.
:: Synchronizing package databases... core 104.4 KiB 104K/s 00:01 [######################] 100% extra 1416.9 KiB 171K/s 00:08 [######################] 100% community 1916.6 KiB 204K/s 00:09 [######################] 100%
It is 3.5M it total even if only just a few new packages were added. Multiply 3.5M to number of users and you'll get a huge number. IMHO it is waste of bandwidth and user time. From other side index is incremental by it nature. Could pacman use index format that fits incremental updates better?
A naive proposal is to use compressed format for "full fetch" (e.g. for users who did not update more than a week), and uncompressed append-only index for incremental. On incremental update client sends request to server "give me append file file starting from position XXX till the end of file", so only delta from the last update will be fetched. Then pacman applies the delta to client index.
What do you think about this idea?
Ok, found pacman.conf option "UseDelta" that seems does what I need. BTW why it is not enabled by default? Isn't saving bandwidth a good thing?
On Sun, Mar 24, 2013 at 11:26:04AM -0700, Anatol Pomozov wrote:
Hi
On Sun, Mar 24, 2013 at 10:57 AM, Anatol Pomozov <anatol.pomozov@gmail.com> wrote:
Hi,
I am an update junkie. I do "pacman -Suy" several times a day. One thing I do not like with current pacman is that it redownloads whole index file each time I update system.
:: Synchronizing package databases... core 104.4 KiB 104K/s 00:01 [######################] 100% extra 1416.9 KiB 171K/s 00:08 [######################] 100% community 1916.6 KiB 204K/s 00:09 [######################] 100%
It is 3.5M it total even if only just a few new packages were added. Multiply 3.5M to number of users and you'll get a huge number. IMHO it is waste of bandwidth and user time. From other side index is incremental by it nature. Could pacman use index format that fits incremental updates better?
A naive proposal is to use compressed format for "full fetch" (e.g. for users who did not update more than a week), and uncompressed append-only index for incremental. On incremental update client sends request to server "give me append file file starting from position XXX till the end of file", so only delta from the last update will be fetched. Then pacman applies the delta to client index.
What do you think about this idea?
Ok, found pacman.conf option "UseDelta" that seems does what I need. BTW why it is not enabled by default? Isn't saving bandwidth a good thing?
I'm not sure whether "UseDelta" also affects package lists (rather than packages only). Arch Linux does not support deltas on the server side yet, see FS#18590 [1] for details. [1] https://bugs.archlinux.org/task/18590
On Sun, Mar 24, 2013 at 07:28:15PM +0100, Lukas Fleischer wrote:
On Sun, Mar 24, 2013 at 11:26:04AM -0700, Anatol Pomozov wrote:
Hi
On Sun, Mar 24, 2013 at 10:57 AM, Anatol Pomozov <anatol.pomozov@gmail.com> wrote:
Hi,
I am an update junkie. I do "pacman -Suy" several times a day. One thing I do not like with current pacman is that it redownloads whole index file each time I update system.
:: Synchronizing package databases... core 104.4 KiB 104K/s 00:01 [######################] 100% extra 1416.9 KiB 171K/s 00:08 [######################] 100% community 1916.6 KiB 204K/s 00:09 [######################] 100%
It is 3.5M it total even if only just a few new packages were added. Multiply 3.5M to number of users and you'll get a huge number. IMHO it is waste of bandwidth and user time. From other side index is incremental by it nature. Could pacman use index format that fits incremental updates better?
A naive proposal is to use compressed format for "full fetch" (e.g. for users who did not update more than a week), and uncompressed append-only index for incremental. On incremental update client sends request to server "give me append file file starting from position XXX till the end of file", so only delta from the last update will be fetched. Then pacman applies the delta to client index.
What do you think about this idea?
Ok, found pacman.conf option "UseDelta" that seems does what I need. BTW why it is not enabled by default? Isn't saving bandwidth a good thing?
I'm not sure whether "UseDelta" also affects package lists (rather than packages only). Arch Linux does not support deltas on the server side yet, see FS#18590 [1] for details.
UseDelta only affects packages. It's simply not feasible to create deltas for the sync DBs as they exist today, especially over HTTP/FTP.
On 25/03/13 04:39, Dave Reisner wrote:
On Sun, Mar 24, 2013 at 07:28:15PM +0100, Lukas Fleischer wrote:
On Sun, Mar 24, 2013 at 11:26:04AM -0700, Anatol Pomozov wrote:
Hi
On Sun, Mar 24, 2013 at 10:57 AM, Anatol Pomozov <anatol.pomozov@gmail.com> wrote:
Hi,
I am an update junkie. I do "pacman -Suy" several times a day. One thing I do not like with current pacman is that it redownloads whole index file each time I update system.
:: Synchronizing package databases... core 104.4 KiB 104K/s 00:01 [######################] 100% extra 1416.9 KiB 171K/s 00:08 [######################] 100% community 1916.6 KiB 204K/s 00:09 [######################] 100%
It is 3.5M it total even if only just a few new packages were added. Multiply 3.5M to number of users and you'll get a huge number. IMHO it is waste of bandwidth and user time. From other side index is incremental by it nature. Could pacman use index format that fits incremental updates better?
A naive proposal is to use compressed format for "full fetch" (e.g. for users who did not update more than a week), and uncompressed append-only index for incremental. On incremental update client sends request to server "give me append file file starting from position XXX till the end of file", so only delta from the last update will be fetched. Then pacman applies the delta to client index.
What do you think about this idea?
Ok, found pacman.conf option "UseDelta" that seems does what I need. BTW why it is not enabled by default? Isn't saving bandwidth a good thing?
I'm not sure whether "UseDelta" also affects package lists (rather than packages only). Arch Linux does not support deltas on the server side yet, see FS#18590 [1] for details.
UseDelta only affects packages. It's simply not feasible to create deltas for the sync DBs as they exist today, especially over HTTP/FTP.
To clarify - 10s if not 100s of packages get updated on an average day. That is a lot of deltas...
On Tue, Mar 26, 2013 at 01:56:35PM +1000, Allan McRae wrote:
On 25/03/13 04:39, Dave Reisner wrote:
On Sun, Mar 24, 2013 at 07:28:15PM +0100, Lukas Fleischer wrote:
On Sun, Mar 24, 2013 at 11:26:04AM -0700, Anatol Pomozov wrote:
Hi
On Sun, Mar 24, 2013 at 10:57 AM, Anatol Pomozov <anatol.pomozov@gmail.com> wrote:
Hi,
I am an update junkie. I do "pacman -Suy" several times a day. One thing I do not like with current pacman is that it redownloads whole index file each time I update system.
:: Synchronizing package databases... core 104.4 KiB 104K/s 00:01 [######################] 100% extra 1416.9 KiB 171K/s 00:08 [######################] 100% community 1916.6 KiB 204K/s 00:09 [######################] 100%
It is 3.5M it total even if only just a few new packages were added. Multiply 3.5M to number of users and you'll get a huge number. IMHO it is waste of bandwidth and user time. From other side index is incremental by it nature. Could pacman use index format that fits incremental updates better?
A naive proposal is to use compressed format for "full fetch" (e.g. for users who did not update more than a week), and uncompressed append-only index for incremental. On incremental update client sends request to server "give me append file file starting from position XXX till the end of file", so only delta from the last update will be fetched. Then pacman applies the delta to client index.
What do you think about this idea?
Ok, found pacman.conf option "UseDelta" that seems does what I need. BTW why it is not enabled by default? Isn't saving bandwidth a good thing?
I'm not sure whether "UseDelta" also affects package lists (rather than packages only). Arch Linux does not support deltas on the server side yet, see FS#18590 [1] for details.
UseDelta only affects packages. It's simply not feasible to create deltas for the sync DBs as they exist today, especially over HTTP/FTP.
To clarify - 10s if not 100s of packages get updated on an average day. That is a lot of deltas...
If only libgit2 was still a thing... -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF
participants (5)
-
Allan McRae
-
Anatol Pomozov
-
Dave Reisner
-
Lukas Fleischer
-
William Giokas