[pacman-dev] makepkg integrity check patches
Here is some justification for the patches on my working branch related to integrity checking. Put in order from least controversial to most... 1) cd3910b makepkg -g: use checksums defined in the pkgbuild This is Xavier's patch. I am not sure there was ever issues with this one, but I think it got caught up with the other patches. This patch just makes it easier to maintain checksums in a PKGBUILD other than the default in makepkg.conf. e.g. if I have sha1sums in the PKGBUILD, "makepkg -g" will generate sha1sums even if generating md5sums is the default. To generate md5sums only, you just need to remove the old checksums from the PKGBUILD or add an empty md5sum array to the PKGBUILD. 2) cd1378d makepkg: rework --skipinteg This is very, very, VERY useful. I did not have makepkg-git on my new computer earlier this week and the current makepkg behaviour annoyed me A LOT. without patch:
/usr/bin/makepkg --skipinteg ... ==> Validating source files with md5sums... abs-2.3.4.1.tar.gz ... FAILED ==> ERROR: One or more files did not pass the validity check!
with patch:
makepkg --skipinteg ... ==> WARNING: Skipping integrity checks. ==> Extracting Sources...
This is particularly useful when testing out a patch that you need to repeatedly modify. You only need to update your checksums once it is working. I use this very frequently, but then again I do more packaging than most. 3) 5d911ae makepkg: allow skipping integrity checks when making source package And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed. Admittedly, this patch will see little use. I used to use it in combination with #2 when checking that a modified patch compiled on both my i686 and x86_64 machines. Now that I build both architectures on one machine, this is of less use to me. I have also used it when commenting on a PKGBUILD that was sent to me with a query about the best way to implement something. There I modified some of the build() function to be clearer and sent it to the author saying I would use this approach but it is not tested. Note I could have used tar manually (in fact the tar file name would have been tab autocompleted so the command would probably be simpler to type), but then I would not get the other checks makepkg provides. Note that this patch does not make --skipinteg the default. In fact, given how long the options is, it is very unlikely that a user could every accidentally type it! There is no difference in behaviour to the current "makepkg --source" unless the user specifically requests it. The patch just makes --skipinteg do what its name says; skip integrity checks. Allan
On Wed, May 5, 2010 at 7:51 PM, Allan McRae <allan@archlinux.org> wrote:
Here is some justification for the patches on my working branch related to integrity checking. Put in order from least controversial to most...
1) cd3910b makepkg -g: use checksums defined in the pkgbuild
This is Xavier's patch. I am not sure there was ever issues with this one, but I think it got caught up with the other patches. This patch just makes it easier to maintain checksums in a PKGBUILD other than the default in makepkg.conf. e.g. if I have sha1sums in the PKGBUILD, "makepkg -g" will generate sha1sums even if generating md5sums is the default. To generate md5sums only, you just need to remove the old checksums from the PKGBUILD or add an empty md5sum array to the PKGBUILD.
Already merged, I have zero problems here.
2) cd1378d makepkg: rework --skipinteg
This is very, very, VERY useful. I did not have makepkg-git on my new computer earlier this week and the current makepkg behaviour annoyed me A LOT.
without patch:
/usr/bin/makepkg --skipinteg ... ==> Validating source files with md5sums... abs-2.3.4.1.tar.gz ... FAILED ==> ERROR: One or more files did not pass the validity check!
with patch:
makepkg --skipinteg ... ==> WARNING: Skipping integrity checks. ==> Extracting Sources...
This is particularly useful when testing out a patch that you need to repeatedly modify. You only need to update your checksums once it is working. I use this very frequently, but then again I do more packaging than most.
Fine with me, and also merged locally.
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
Part of me says this is at least a barrier they would have to work to cross and defeat, and just downloading the darn source would be quicker for most. I do understand that a quick `tar czf` would circumvent the whole thing, but it has become so easy to create source packages that I don't feel many people even think about it this way anymore. Correct me if I'm wrong.
Admittedly, this patch will see little use. I used to use it in combination with #2 when checking that a modified patch compiled on both my i686 and x86_64 machines. Now that I build both architectures on one machine, this is of less use to me. I have also used it when commenting on a PKGBUILD that was sent to me with a query about the best way to implement something. There I modified some of the build() function to be clearer and sent it to the author saying I would use this approach but it is not tested. Note I could have used tar manually (in fact the tar file name would have been tab autocompleted so the command would probably be simpler to type), but then I would not get the other checks makepkg provides.
Note that this patch does not make --skipinteg the default. In fact, given how long the options is, it is very unlikely that a user could every accidentally type it! There is no difference in behaviour to the current "makepkg --source" unless the user specifically requests it. The patch just makes --skipinteg do what its name says; skip integrity checks.
Well after reading most of this you may have won me over. Maybe I can get just one small concession- when you use both of these options together (or --allsource I believe it is), we print one more warning saying "this source package is not guaranteed to work for anyone else!" or something along those lines. -Dan
On 06/05/10 11:10, Dan McGee wrote:
Well after reading most of this you may have won me over. Maybe I can get just one small concession- when you use both of these options together (or --allsource I believe it is), we print one more warning saying "this source package is not guaranteed to work for anyone else!" or something along those lines.
Sure. How about changing (or comibinig): warning "$(gettext "Skipping integrity checks.")" to warning "$(gettext "Perfoming integrity checks is recommended when creating source packages for distribution.")" plain "$(gettext "Skipping integrity checks contributes to global warming, terrorism, and most of the worlds other problems.")" Maybe not the second half... Allan
On 06/05/10 11:10, Dan McGee wrote:
On Wed, May 5, 2010 at 7:51 PM, Allan McRae<allan@archlinux.org> wrote:
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
Part of me says this is at least a barrier they would have to work to cross and defeat, and just downloading the darn source would be quicker for most. I do understand that a quick `tar czf` would circumvent the whole thing, but it has become so easy to create source packages that I don't feel many people even think about it this way anymore. Correct me if I'm wrong.
Admittedly, this patch will see little use. I used to use it in combination with #2 when checking that a modified patch compiled on both my i686 and x86_64 machines. Now that I build both architectures on one machine, this is of less use to me. I have also used it when commenting on a PKGBUILD that was sent to me with a query about the best way to implement something. There I modified some of the build() function to be clearer and sent it to the author saying I would use this approach but it is not tested. Note I could have used tar manually (in fact the tar file name would have been tab autocompleted so the command would probably be simpler to type), but then I would not get the other checks makepkg provides.
Note that this patch does not make --skipinteg the default. In fact, given how long the options is, it is very unlikely that a user could every accidentally type it! There is no difference in behaviour to the current "makepkg --source" unless the user specifically requests it. The patch just makes --skipinteg do what its name says; skip integrity checks.
Well after reading most of this you may have won me over. Maybe I can get just one small concession- when you use both of these options together (or --allsource I believe it is), we print one more warning saying "this source package is not guaranteed to work for anyone else!" or something along those lines.
I have pushed a more strongly worded message into my working branch. I am sure the message could be improved. Allan
On Thu 06 May 2010 10:51 +1000, Allan McRae wrote:
2) cd1378d makepkg: rework --skipinteg
This is very, very, VERY useful. I did not have makepkg-git on my new computer earlier this week and the current makepkg behaviour annoyed me A LOT.
This is particularly useful when testing out a patch that you need to repeatedly modify. You only need to update your checksums once it is working. I use this very frequently, but then again I do more packaging than most.
I believe this is bad behaviour. makepkg should be used to package software, not help you develop patches for it.
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
Just because someone can manually make a bad source package there's no excuse to put bad behaviour into makepkg. The same applies to binary packages. Perhaps in the future if package signing is implemented for packages it would also be possible to have signed source packages. I think package integrity is most important when using these official tools. If shortcuts are required they should be developed elsewhere.
On 06/05/10 11:41, Loui Chang wrote:
On Thu 06 May 2010 10:51 +1000, Allan McRae wrote:
2) cd1378d makepkg: rework --skipinteg
This is very, very, VERY useful. I did not have makepkg-git on my new computer earlier this week and the current makepkg behaviour annoyed me A LOT.
This is particularly useful when testing out a patch that you need to repeatedly modify. You only need to update your checksums once it is working. I use this very frequently, but then again I do more packaging than most.
I believe this is bad behaviour. makepkg should be used to package software, not help you develop patches for it.
Not being condescending here, but you obviously do not do much packaging. Packaging software requires patching software. e.g. gcc-4.x header changes, libpng API changes, etc. It is a lot easier for me to run "makepkg --skipinteg" to test the state of a patch to fix build issues that it is to manually extract the tarball, apply the patch, configure, make...
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
Just because someone can manually make a bad source package there's no excuse to put bad behaviour into makepkg. The same applies to binary packages.
Why is it bad behaviour? I think you are just assuming the user is stupid and will use it unnecessarily. "pacman -Rd" and "pacman -Sf" are stupid in most cases, but we do not remove them as they are also useful in others cases. Similarly, I provided two usage cases where it is perfectly reasonable behaviour to skip integrity checks. Skipping integrity checks is not going to be the default behaviour and does not even have a shorthand option. The user has to specifically want to use it. Lets make the assumption that if someone goes out of their way to type "--skipinteg", that they are doing it deliberately.
Perhaps in the future if package signing is implemented for packages it would also be possible to have signed source packages.
Yes it would, but entirely off topic.
On Thu, May 6, 2010 at 4:09 AM, Allan McRae <allan@archlinux.org> wrote:
On 06/05/10 11:41, Loui Chang wrote:
On Thu 06 May 2010 10:51 +1000, Allan McRae wrote:
2) cd1378d makepkg: rework --skipinteg
This is very, very, VERY useful. I did not have makepkg-git on my new computer earlier this week and the current makepkg behaviour annoyed me A LOT.
This is particularly useful when testing out a patch that you need to repeatedly modify. You only need to update your checksums once it is working. I use this very frequently, but then again I do more packaging than most.
I believe this is bad behaviour. makepkg should be used to package software, not help you develop patches for it.
Not being condescending here, but you obviously do not do much packaging. Packaging software requires patching software. e.g. gcc-4.x header changes, libpng API changes, etc. It is a lot easier for me to run "makepkg --skipinteg" to test the state of a patch to fix build issues that it is to manually extract the tarball, apply the patch, configure, make...
To complete Allan's arguments : 90% of the pkgbuilds I build are not meant to be shared with anyone else, usually ABS or AUR packages, sometimes slightly customized for my needs. Why couldn't I enable a non-default behavior that skip integs when I know they will fail and I don't care what the new checksums are ? "non-default behavior" is the important part here, if this was default, I would be totally against this behavior.
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
Just because someone can manually make a bad source package there's no excuse to put bad behaviour into makepkg. The same applies to binary packages.
Why is it bad behaviour? I think you are just assuming the user is stupid and will use it unnecessarily. "pacman -Rd" and "pacman -Sf" are stupid in most cases, but we do not remove them as they are also useful in others cases. Similarly, I provided two usage cases where it is perfectly reasonable behaviour to skip integrity checks.
Skipping integrity checks is not going to be the default behaviour and does not even have a shorthand option. The user has to specifically want to use it. Lets make the assumption that if someone goes out of their way to type "--skipinteg", that they are doing it deliberately.
Agreed. If we assume the user is completely stupid, we can't develop softwares. We can't produce anything actually. We might need a notice or disclaimer to ship pacman and makepkg :P
On Thu 06 May 2010 12:09 +1000, Allan McRae wrote:
On 06/05/10 11:41, Loui Chang wrote:
On Thu 06 May 2010 10:51 +1000, Allan McRae wrote:
2) cd1378d makepkg: rework --skipinteg
This is very, very, VERY useful. I did not have makepkg-git on my new computer earlier this week and the current makepkg behaviour annoyed me A LOT.
This is particularly useful when testing out a patch that you need to repeatedly modify. You only need to update your checksums once it is working. I use this very frequently, but then again I do more packaging than most.
I believe this is bad behaviour. makepkg should be used to package software, not help you develop patches for it.
Not being condescending here, but you obviously do not do much packaging. Packaging software requires patching software. e.g. gcc-4.x header changes, libpng API changes, etc. It is a lot easier for me to run "makepkg --skipinteg" to test the state of a patch to fix build issues that it is to manually extract the tarball, apply the patch, configure, make...
I understand that, but it's not very efficient, nor does it usually make much sense to fully package something just to test a patch. I've done enough devel/patching to discover that I think. You can skip part of build, multiple extraction, copies, and compression. This has a particular impact with very large packages, not so much with small ones.
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
Just because someone can manually make a bad source package there's no excuse to put bad behaviour into makepkg. The same applies to binary packages.
Why is it bad behaviour? I think you are just assuming the user is stupid and will use it unnecessarily. "pacman -Rd" and "pacman -Sf" are stupid in most cases, but we do not remove them as they are also useful in others cases. Similarly, I provided two usage cases where it is perfectly reasonable behaviour to skip integrity checks.
Skipping integrity checks is not going to be the default behaviour and does not even have a shorthand option. The user has to specifically want to use it. Lets make the assumption that if someone goes out of their way to type "--skipinteg", that they are doing it deliberately.
Perhaps in the future if package signing is implemented for packages it would also be possible to have signed source packages.
Yes it would, but entirely off topic.
This relates to package integrity. I guess I mean to present the odd possibility where you trust the person who signed the package, but the it hasn't even passed basic integrity checks. I guess the debate is convenience versus correctness really. I can understand if someone may value the convenience more, but I contend that the gained convenience is not particularly valuable after all, can be obtained in other ways, and should not be put into the official tools at the potential sacrifice of correctness.
On Thu, May 6, 2010 at 12:50 PM, Loui Chang <louipc.ist@gmail.com> wrote:
This relates to package integrity. I guess I mean to present the odd possibility where you trust the person who signed the package, but the it hasn't even passed basic integrity checks.
I guess the debate is convenience versus correctness really.
No, it's not, we want both. default behavior -> correctness non-default behavior for people who know what they are doing -> convenience Very much like pacman -Sd / -Sf as Allan already said multiple times.
I can understand if someone may value the convenience more, but I contend that the gained convenience is not particularly valuable after all, can be obtained in other ways, and should not be put into the official tools at the potential sacrifice of correctness.
The only sacrifice we will make is packagers who dare sharing a pkgbuild with wrong checksums. Allan told me he will burn them all on the public place. Just like we would do with people that would send a pkgbuild with rm -rf / inside.
On Thu 06 May 2010 12:58 +0200, Xavier Chantry wrote:
On Thu, May 6, 2010 at 12:50 PM, Loui Chang <louipc.ist@gmail.com> wrote:
This relates to package integrity. I guess I mean to present the odd possibility where you trust the person who signed the package, but the it hasn't even passed basic integrity checks.
I guess the debate is convenience versus correctness really.
No, it's not, we want both. default behavior -> correctness non-default behavior for people who know what they are doing -> convenience Very much like pacman -Sd / -Sf as Allan already said multiple times.
As for analogies, I'm thinking it's more like an option for an HTML generator to produce flawed markup to display nicely in a crappy browser. Supporting bad behaviour is bad.
I can understand if someone may value the convenience more, but I contend that the gained convenience is not particularly valuable after all, can be obtained in other ways, and should not be put into the official tools at the potential sacrifice of correctness.
The only sacrifice we will make is packagers who dare sharing a pkgbuild with wrong checksums. Allan told me he will burn them all on the public place.
Hah. I think he said that he does share them. Anyways. I do at least believe it should be possible to do programmatically, thus makepkg should provide the functions for skipinteg. Maybe it could be a hidden, undocumented option. I'd be a lot more comfortable with that.
On Thu, 2010-05-06 at 10:51 +1000, Allan McRae wrote:
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
I found a use case for this recently. For some reason uploading the tarball of my project to GitHub changed its checksum, so had to adjust that in the PKGBUILD. But when I put it on the AUR, people complained that the checksum was wrong. I tried to revert it, but makepkg would not let me run --source without the original tarball (which I had deleted), so I had to run make dist all over again, re-upload and so on. This time I used the original checksum (after checking that the extracted tarballs were the same, of course), and that seemed to work. But it would have been easier for me if makepkg just skipped the --source integrity check. Jonathan
On Thu 06 May 2010 15:59 +1200, Jonathan Conder wrote:
On Thu, 2010-05-06 at 10:51 +1000, Allan McRae wrote:
3) 5d911ae makepkg: allow skipping integrity checks when making source package
And here is the fun one... "makepkg --source" currently requires checking all checksums. Using "-source --skipinteg" does not skip this, which in itself makes little sense to me. The argument that this stops people distributing packages with bad checksums is flawed. There is nothing stopping them doing that now. They just have to not use makepkg when creating the tarball, which could lead to even worse PKGBUILDs being distributed as none of makepkg's other checks would be performed.
I found a use case for this recently. For some reason uploading the tarball of my project to GitHub changed its checksum, so had to adjust that in the PKGBUILD. But when I put it on the AUR, people complained that the checksum was wrong. I tried to revert it, but makepkg would not let me run --source without the original tarball (which I had deleted), so I had to run make dist all over again, re-upload and so on. This time I used the original checksum (after checking that the extracted tarballs were the same, of course), and that seemed to work. But it would have been easier for me if makepkg just skipped the --source integrity check.
I think checksums were implemented for exactly that type of situation. If the server altered your file, or there was some server error, then the check should fail.
participants (5)
-
Allan McRae
-
Dan McGee
-
Jonathan Conder
-
Loui Chang
-
Xavier Chantry