[pacman-dev] Delta testing
I have created a repo for those you who may want to help test the xdelta code in pacman 3.1. There is just one package in the repo: deltatest. I will try to update the package at least once a day. Steps for delta goodness: 1. Install pacman from git 2. Install xdelta: pacman -S xdelta 3. Add UseDelta to your pacman.conf 4. Add the deltatest server to pacman.conf: [deltatest] Server = http://home.insightbb.com/~nathanj/dt 5. Install the package: pacman -Sy deltatest 6. Update the package whenever you want 7. Report many, many bugs
On Wed, Nov 07, 2007 at 05:02:01PM -0500, Nathan Jones wrote:
I have created a repo for those you who may want to help test the xdelta code in pacman 3.1. There is just one package in the repo: deltatest. I will try to update the package at least once a day.
Steps for delta goodness:
1. Install pacman from git 2. Install xdelta: pacman -S xdelta 3. Add UseDelta to your pacman.conf 4. Add the deltatest server to pacman.conf:
[deltatest] Server = http://home.insightbb.com/~nathanj/dt
5. Install the package: pacman -Sy deltatest 6. Update the package whenever you want 7. Report many, many bugs
I did these steps, I guess I have to wait for tomorrow to see if it works :) Btw, already noticed a little problem not related to delta, but I don't know what causes it : $ sudo pacman -Sy --debug | grep mtime [23:10:46] debug: sync: new mtime for deltatest: 20071107230659 [23:10:46] debug: mtimes are identical, skipping testing.db.tar.gz $ sudo pacman -Sy --debug | grep mtime [23:10:49] debug: sync: new mtime for deltatest: 20071107230702 [23:10:49] debug: mtimes are identical, skipping testing.db.tar.gz That is, I get a different mtime from your server on every -Sy. Other mirrors work fine, using ftp or http. Not sure what is in fault here.
On Wed, Nov 07, 2007 at 11:15:57PM +0100, Xavier wrote:
I did these steps, I guess I have to wait for tomorrow to see if it works :)
Btw, already noticed a little problem not related to delta, but I don't know what causes it : $ sudo pacman -Sy --debug | grep mtime [23:10:46] debug: sync: new mtime for deltatest: 20071107230659 [23:10:46] debug: mtimes are identical, skipping testing.db.tar.gz $ sudo pacman -Sy --debug | grep mtime [23:10:49] debug: sync: new mtime for deltatest: 20071107230702 [23:10:49] debug: mtimes are identical, skipping testing.db.tar.gz
That is, I get a different mtime from your server on every -Sy. Other mirrors work fine, using ftp or http. Not sure what is in fault here.
The mtime seems to have stabilized at 22:37:22 GMT, but I am not sure why. I uploaded another file to test, and I get different mtimes every download like you noticed. The mtime for the test file as seen in the ftp client is 'Wed Nov 7 00:00:00 2007'. gftp sends the command 'site utime' after uploading a file, but the server does not recognize it. So I don't know!
On Wed, Nov 07, 2007 at 06:41:34PM -0500, Nathan Jones wrote:
The mtime seems to have stabilized at 22:37:22 GMT, but I am not sure why. I uploaded another file to test, and I get different mtimes every download like you noticed. The mtime for the test file as seen in the ftp client is 'Wed Nov 7 00:00:00 2007'. gftp sends the command 'site utime' after uploading a file, but the server does not recognize it.
So I don't know!
Well, it seems to work now. Also the first delta based upgrade I did this morning was apparently a success :)
On Nov 8, 2007 6:50 AM, Xavier <shiningxc@gmail.com> wrote:
On Wed, Nov 07, 2007 at 06:41:34PM -0500, Nathan Jones wrote:
The mtime seems to have stabilized at 22:37:22 GMT, but I am not sure why. I uploaded another file to test, and I get different mtimes every download like you noticed. The mtime for the test file as seen in the ftp client is 'Wed Nov 7 00:00:00 2007'. gftp sends the command 'site utime' after uploading a file, but the server does not recognize it.
So I don't know!
Well, it seems to work now.
Also the first delta based upgrade I did this morning was apparently a success :)
I delta-upgraded pacman-git last night...success! Well I will report one small but unlikely issue- I had an old version of 20071105 in my cache, so it had the wrong MD5sum. Perhaps the code should check for this before it does any delta-patching instead of waiting until the patching is done before checking all MD5sums? It took me a little while to figure out exactly what the error message was saying and what was wrong. -Dan
On Nov 7, 2007 4:02 PM, Nathan Jones <nathanj@insightbb.com> wrote:
I have created a repo for those you who may want to help test the xdelta code in pacman 3.1. There is just one package in the repo: deltatest. I will try to update the package at least once a day.
Steps for delta goodness:
1. Install pacman from git 2. Install xdelta: pacman -S xdelta 3. Add UseDelta to your pacman.conf 4. Add the deltatest server to pacman.conf:
[deltatest] Server = http://home.insightbb.com/~nathanj/dt
5. Install the package: pacman -Sy deltatest 6. Update the package whenever you want 7. Report many, many bugs
Awesome! I would be willing to turn my [pacman-git] repository into one with deltas as well (as I will probably rebuild this any time we check changes into master), should we try that? Nathan, can you give me pointers as to what exactly I need to do to get this done? Is it anything more than adding the xdelta option? Speaking of the xdelta option- can I add that to either makepkg.conf OR the package itself? I know some of our other options are like this, and it would be a cool feature to say "never use xdelta on this package" with options=(!xdelta). Or in this case, since I don't want to build all my packages with deltas, can I tell it to just build this one by using options=(xdelta)? -Dan
On Wed, Nov 07, 2007 at 05:55:50PM -0600, Dan McGee wrote:
Awesome! I would be willing to turn my [pacman-git] repository into one with deltas as well (as I will probably rebuild this any time we check changes into master), should we try that? Nathan, can you give me pointers as to what exactly I need to do to get this done? Is it anything more than adding the xdelta option?
Adding xdelta should be all that is needed. repo-add will automatically look in the current directory for any delta files and add them.
Speaking of the xdelta option- can I add that to either makepkg.conf OR the package itself? I know some of our other options are like this, and it would be a cool feature to say "never use xdelta on this package" with options=(!xdelta). Or in this case, since I don't want to build all my packages with deltas, can I tell it to just build this one by using options=(xdelta)?
Just tested this, but it looks like it always follows the option in makepkg.conf. I will look into making this configurable per package.
Setting xdelta (or !xdelta) in the options array in a PKGBUILD will override the makepkg.conf setting. Signed-off-by: Nathan Jones <nathanj@insightbb.com> --- scripts/makepkg.sh.in | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index f3e4741..a8d7747 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -868,7 +868,9 @@ create_package() { } create_xdelta() { - if [ "$(check_buildenv xdelta)" != "y" ]; then + if [ "$(check_option xdelta)" = "n" ]; then + return + elif [ "$(check_buildenv xdelta)" != "y" -a "$(check_option xdelta)" != "y" ]; then return elif [ ! "$(type -p xdelta)" ]; then error "$(gettext "Cannot find the xdelta binary! Is xdelta installed?")" -- 1.5.3.5
On Nov 8, 2007 1:37 PM, Nathan Jones <nathanj@insightbb.com> wrote:
Setting xdelta (or !xdelta) in the options array in a PKGBUILD will override the makepkg.conf setting.
So this means we can generate deltas on a package by package basis, like for OpenOffice or the like, correct? The reason I ask is... well, if it's per-package now, do we really need a buildenv option for this as well?
On Thu, Nov 08, 2007 at 01:46:34PM -0600, Aaron Griffin wrote:
On Nov 8, 2007 1:37 PM, Nathan Jones <nathanj@insightbb.com> wrote:
Setting xdelta (or !xdelta) in the options array in a PKGBUILD will override the makepkg.conf setting.
So this means we can generate deltas on a package by package basis, like for OpenOffice or the like, correct?
The reason I ask is... well, if it's per-package now, do we really need a buildenv option for this as well?
If a person maintains 100 packages and decides to start creating deltas, I would think he would prefer to add xdelta to makepkg.conf rather than to 100 different PKGBUILDs. In regards to the patch, I looked back through makepkg and noticed that check_option checks the PKGBUILD first then makepkg.conf. I was thinking that check_option only checked the PKGBUILD. I might resubmit the patch after I test it.
On Nov 8, 2007 3:36 PM, Nathan Jones <nathanj@insightbb.com> wrote:
On Thu, Nov 08, 2007 at 01:46:34PM -0600, Aaron Griffin wrote:
On Nov 8, 2007 1:37 PM, Nathan Jones <nathanj@insightbb.com> wrote:
Setting xdelta (or !xdelta) in the options array in a PKGBUILD will override the makepkg.conf setting.
So this means we can generate deltas on a package by package basis, like for OpenOffice or the like, correct?
The reason I ask is... well, if it's per-package now, do we really need a buildenv option for this as well?
If a person maintains 100 packages and decides to start creating deltas, I would think he would prefer to add xdelta to makepkg.conf rather than to 100 different PKGBUILDs.
In regards to the patch, I looked back through makepkg and noticed that check_option checks the PKGBUILD first then makepkg.conf. I was thinking that check_option only checked the PKGBUILD. I might resubmit the patch after I test it.
When I suggested this, I was thinking of the following type code that is already in makepkg (line 659): # use ccache if it is requested (check buildenv and PKGBUILD opts) if [ "$(check_buildenv ccache)" = "y" -a "$(check_option ccache)" != "n" ]; then [ -d /usr/lib/ccache/bin ] && export PATH="/usr/lib/ccache/bin:$PATH" fi Note that ccache is a BUILDENV var in makepkg.conf, but can still be denied in a PKGBUILD by using !ccache (for those packages that don't build correctly with it). I feel like a if clause just like the above would work for xdelta as well (and cut your patch from -1/+3 to -1/+1). -Dan
On Thu, Nov 08, 2007 at 03:50:12PM -0600, Dan McGee wrote:
On Nov 8, 2007 3:36 PM, Nathan Jones <nathanj@insightbb.com> wrote:
On Thu, Nov 08, 2007 at 01:46:34PM -0600, Aaron Griffin wrote:
On Nov 8, 2007 1:37 PM, Nathan Jones <nathanj@insightbb.com> wrote:
Setting xdelta (or !xdelta) in the options array in a PKGBUILD will override the makepkg.conf setting.
So this means we can generate deltas on a package by package basis, like for OpenOffice or the like, correct?
The reason I ask is... well, if it's per-package now, do we really need a buildenv option for this as well?
If a person maintains 100 packages and decides to start creating deltas, I would think he would prefer to add xdelta to makepkg.conf rather than to 100 different PKGBUILDs.
I think I misunderstood you. Are you suggesting to move the xdelta option from BUILDENV array to the OPTIONS array? If so, I think that is a good idea.
In regards to the patch, I looked back through makepkg and noticed that check_option checks the PKGBUILD first then makepkg.conf. I was thinking that check_option only checked the PKGBUILD. I might resubmit the patch after I test it.
When I suggested this, I was thinking of the following type code that is already in makepkg (line 659): # use ccache if it is requested (check buildenv and PKGBUILD opts) if [ "$(check_buildenv ccache)" = "y" -a "$(check_option ccache)" != "n" ]; then [ -d /usr/lib/ccache/bin ] && export PATH="/usr/lib/ccache/bin:$PATH" fi
Note that ccache is a BUILDENV var in makepkg.conf, but can still be denied in a PKGBUILD by using !ccache (for those packages that don't build correctly with it). I feel like a if clause just like the above would work for xdelta as well (and cut your patch from -1/+3 to -1/+1).
I kept getting confused while working through the different 'y', 'n', and '?' combinations which led to my if statement. If the xdelta option stays in the BUILDENV array I will copy that line.
On Nov 8, 2007 4:10 PM, Nathan Jones <nathanj@insightbb.com> wrote:
On Thu, Nov 08, 2007 at 03:50:12PM -0600, Dan McGee wrote:
On Nov 8, 2007 3:36 PM, Nathan Jones <nathanj@insightbb.com> wrote:
On Thu, Nov 08, 2007 at 01:46:34PM -0600, Aaron Griffin wrote:
On Nov 8, 2007 1:37 PM, Nathan Jones <nathanj@insightbb.com> wrote:
Setting xdelta (or !xdelta) in the options array in a PKGBUILD will override the makepkg.conf setting.
So this means we can generate deltas on a package by package basis, like for OpenOffice or the like, correct?
The reason I ask is... well, if it's per-package now, do we really need a buildenv option for this as well?
If a person maintains 100 packages and decides to start creating deltas, I would think he would prefer to add xdelta to makepkg.conf rather than to 100 different PKGBUILDs.
I think I misunderstood you. Are you suggesting to move the xdelta option from BUILDENV array to the OPTIONS array? If so, I think that is a good idea.
Here is how the two arrays were meant to work (I think): BUILDENV- these are things that affect the building of the package, but NOT the package contents (ccache or distcc should in theory not affect the compiled product). OPTIONS- these are things that affect the actual package contents (removing man pages and stripping binaries produce a different package than one built without those options). So with that out there, I think xdelta should stay in BUILDENV. Apparantly we made a decision a while back to not have both an options=() and a buildenv=() array in the PKGBUILD, so those options and environment settings are all smashed into one array. -Dan
On Thu, Nov 08, 2007 at 03:50:12PM -0600, Dan McGee wrote:
When I suggested this, I was thinking of the following type code that is already in makepkg (line 659): # use ccache if it is requested (check buildenv and PKGBUILD opts) if [ "$(check_buildenv ccache)" = "y" -a "$(check_option ccache)" != "n" ]; then [ -d /usr/lib/ccache/bin ] && export PATH="/usr/lib/ccache/bin:$PATH" fi
Note that ccache is a BUILDENV var in makepkg.conf, but can still be denied in a PKGBUILD by using !ccache (for those packages that don't build correctly with it). I feel like a if clause just like the above would work for xdelta as well (and cut your patch from -1/+3 to -1/+1).
After looking at it some more, I don't think this if statement will work for the xdelta option. This is because the ccache statement only allows the option to be disabled, not enabled, in the PKGBUILD. Consider the scenario where you have !ccache is makepkg.conf, but ccache in PKGBUILD: ccache will _not_ be used. It seems possible to me that some people will disable delta creation for most packages but enable it only for the large ones. It is still possible to turn my if statement into one line, but it will be a long line :). I don't think it would be any better than the patch I've already submitted. Another option would be to do something like this: (untested) local opt=$(check_option xdelta) [ "$opt" = "?" ] && opt=$(check_buildenv xdelta) if [ "$opt" != "y" ]; then return fi Or, check_buildenv could be changed to first look in the PKGBUILD options, like check_option does.
On Nov 8, 2007 6:20 PM, Nathan Jones <nathanj@insightbb.com> wrote:
On Thu, Nov 08, 2007 at 03:50:12PM -0600, Dan McGee wrote:
When I suggested this, I was thinking of the following type code that is already in makepkg (line 659): # use ccache if it is requested (check buildenv and PKGBUILD opts) if [ "$(check_buildenv ccache)" = "y" -a "$(check_option ccache)" != "n" ]; then [ -d /usr/lib/ccache/bin ] && export PATH="/usr/lib/ccache/bin:$PATH" fi
Note that ccache is a BUILDENV var in makepkg.conf, but can still be denied in a PKGBUILD by using !ccache (for those packages that don't build correctly with it). I feel like a if clause just like the above would work for xdelta as well (and cut your patch from -1/+3 to -1/+1).
After looking at it some more, I don't think this if statement will work for the xdelta option. This is because the ccache statement only allows the option to be disabled, not enabled, in the PKGBUILD. Consider the scenario where you have !ccache is makepkg.conf, but ccache in PKGBUILD: ccache will _not_ be used. It seems possible to me that some people will disable delta creation for most packages but enable it only for the large ones.
You definitely bring up some interesting points here. Do we every really want to create deltas for packages under some size limit? Should we make this some kind of configuration setting? I agree with your statement of plausibility as well. My only concern is that currently, an option that is disabled in makepkg.conf cannot be reenabled in a PKGBUILD, which I think is the right way to do things. We don't want only xdelta breaking this rule. However, it would make sense that if someone had no xdelta option at all in their BUILDENV, then a package could choose to turn it on? Mostly thinking aloud here.
It is still possible to turn my if statement into one line, but it will be a long line :). I don't think it would be any better than the patch I've already submitted.
Another option would be to do something like this: (untested)
local opt=$(check_option xdelta) [ "$opt" = "?" ] && opt=$(check_buildenv xdelta)
if [ "$opt" != "y" ]; then return fi
Or, check_buildenv could be changed to first look in the PKGBUILD options, like check_option does.
Hmm. This does mess with the precedence thing I described above, so I don't know. It is one thing for a PKGBUILD to mess with packaging options, but quite another to play with the build environment. -Dan
participants (4)
-
Aaron Griffin
-
Dan McGee
-
Nathan Jones
-
Xavier