[pacman-dev] [PATCH] makepkg: sums: Add FILE and SIGNED-MSG modes
Verifying PGP signatures was only done by expecting them to be detached signature of the source code, which is the case on many occasions. However, some upstreams actually do things a bit differently, specifically there are two other main ways things can be done: - a checksum file is provided, as well as a detached signature of that file. For such cases, this adds a special mode (alongside SKIP) to the sums array: FILE In that case, for filename.ext a file filename.ext.$ALGO is expected to be a standard checksum file - i.e. contain at least a line with the hash, spaces, then the filename - and the hash used will be the one from that file. Obviously this should be used when a detached signature of the file is provided, which will be treated by makepkg just as usual. An example package could be firefox. - a checksum file is provided as a signed message. For such cases, this adds a mode SIGNED-MSG, as it expects a file filename.ext.$ALGO.signed-msg to be the signed message. Upon hash checking, it will "extract" said message from the file via gpg, then use it much like in the FILE case. And during PGP signature checking, *.signed-msg files are verified as signed messages instead of detached signatures. An example package could be harfbuzz. Note that this doesn't change generation (--geninteg) since it can't be done automatically. Signed-off-by: Olivier Brunel <jjk@jjacky.com> --- So the .signed-msg extension is obviously not one that will be found upstream, and will force a rename in the source array. A "common" extension that might be found upstream for such files would be .$ALGO.asc (e.g. .sha1.asc), and using that as well could make the source array a bit "simpler" to write, that is e.g: source=(http://www.freedesktop.org/software/harfbuzz/release/${pkgname}-${pkgver}.tar.bz2{,.sha256.asc}) vs: source=(http://www.freedesktop.org/software/harfbuzz/release/${pkgname}-${pkgver}.tar.bz2) $pkgname-$pkgver.tar.bz2.sha256.signed-msg::http://www.freedesktop.org/software/harfbuzz/release/${pkgname}-${pkgver}.tar.bz2.sha256.asc) I thought of using SIGNED-ASC as mode, and expect such file names, but then in makepkg we can't just identify such files (upon signature checking) simply by names, since it is also a valid name for detached signature of a checksum file (e.g. used in FILE mode), so then it would go: if the source file is missing, and the filename matches *.$ALGO.asc then instead of erroring out process it as a signed message. Felt to me that this wasn't as simple/proper, and so I went with a custom, specific extension. That could be change if you prefer it the other way around, or have a better idea? doc/PKGBUILD.5.txt | 19 +++++++++ scripts/makepkg.sh.in | 104 +++++++++++++++++++++++++++++++++++++------------- 2 files changed, 97 insertions(+), 26 deletions(-) diff --git a/doc/PKGBUILD.5.txt b/doc/PKGBUILD.5.txt index 4ff3b02..4802d2a 100644 --- a/doc/PKGBUILD.5.txt +++ b/doc/PKGBUILD.5.txt @@ -131,6 +131,8 @@ downloaded from version control systems (VCS). For more information, see Files in the source array with extensions `.sig`, `.sign` or, `.asc` are recognized by makepkg as PGP signatures and will be automatically used to verify the integrity of the corresponding source file. +Files with extension `.signed-msg` are treated as signed messages, and verified +as such by makepkg. See below (md5sums) for more on how this can be used. *validpgpkeys (array)*:: An array of PGP fingerprints. If this array is non-empty, makepkg will @@ -153,12 +155,29 @@ contain whitespace characters. in place of a normal hash, the integrity check for that source file will be skipped. To easily generate md5sums, run ``makepkg -g >> PKGBUILD''. If desired, move the md5sums line to an appropriate location. ++ +Two special modes can also be manually put in the array in place of a normal +hash: ++ +- 'FILE' can be used to have makepkg get the hash from a checksum file whose +name shall be the name of the source file, plus extension `.md5` A valid +checksum file is a text file with at least one line with the hash, space(s), +then the filename, much like linkman:md5sum[1] output. ++ +- 'SIGNED-MSG' can be used to have makepkg get the hash from a PGP signed +message whose name shall be the name of the source file, plus extension +`.md5.signed-msg` The actual message is expected to be a checksum file (as with +'FILE'), and the signature will be checked alongside other PGP signatures. *sha1sums, sha256sums, sha384sums, sha512sums (arrays)*:: Alternative integrity checks that makepkg supports; these all behave similar to the md5sums option described above. To enable use and generation of these checksums, be sure to set up the `INTEGRITY_CHECK` option in linkman:makepkg.conf[5]. ++ +Special modes 'FILE' and 'SIGNED-MSG' work the same as described above, only +with the appropriate integrity checks used in file names, e.g. extensions +`.sha1` (with 'FILE') or `.sha512.signed-msg` (with 'SIGNED-MSG') *groups (array)*:: An array of symbolic names that represent groups of packages, allowing diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 22679f7..e4a1ec2 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -209,7 +209,7 @@ source_has_signatures() { get_all_sources_for_arch 'all_sources' for file in "${all_sources[@]}"; do - if [[ ${file%%::*} = *.@(sig?(n)|asc) ]]; then + if [[ ${file%%::*} = *.@(sig?(n)|asc|signed-msg) ]]; then return 0 fi done @@ -380,7 +380,7 @@ generate_one_checksum() { sum="SKIP" ;; *) - if [[ $netfile != *.@(sig?(n)|asc) ]]; then + if [[ $netfile != *.@(sig?(n)|asc|signed-msg) ]]; then local file file="$(get_filepath "$netfile")" || missing_source_file "$netfile" sum="$(openssl dgst -${integ} "$file")" @@ -432,20 +432,67 @@ generate_checksums() { verify_integrity_one() { local source_name=$1 integ=$2 expectedsum=$3 + local sumfile tmp - local file="$(get_filename "$source_name")" - printf ' %s ... ' "$file" >&2 + local filename="$(get_filename "$source_name")" + printf ' %s ... ' "$filename" >&2 if [[ $expectedsum = 'SKIP' ]]; then printf '%s\n' "$(gettext "Skipped")" >&2 return fi - if ! file="$(get_filepath "$file")"; then + if ! file="$(get_filepath "$filename")"; then printf '%s\n' "$(gettext "NOT FOUND")" >&2 return 1 fi + if [[ $expectedsum = 'SIGNED-MSG' ]]; then + sumfile="$file.$integ.signed-msg" + if [ ! -e "$sumfile" ]; then + printf '%s: ' "$(gettext "FAILED")" >&2 + printf "$(gettext "%s missing")" "$(get_filename "$sumfile")" >&2 + printf '\n' >&2 + return 1 + fi + tmp=$(mktemp --dry-run) + gpg --quiet --batch --skip-verify --output "$tmp" "$sumfile" 2> /dev/null + if [ $? -ne 0 ]; then + printf '%s: ' "$(gettext "FAILED")" >&2 + printf "$(gettext "%s invalid")" "$(get_filename "$sumfile")" >&2 + printf '\n' >&2 + return 1 + fi + expectedsum='FILE' + fi + + if [[ $expectedsum = 'FILE' ]]; then + if [ -z "$tmp" ]; then + sumfile="$file.$integ" + else + sumfile="$tmp" + fi + if [ ! -e "$sumfile" ]; then + printf '%s: ' "$(gettext "FAILED")" >&2 + printf "$(gettext "%s missing")" "$(get_filename "$sumfile")" >&2 + printf '\n' >&2 + [ -e "$tmp" ] && rm -f "$tmp" + return 1 + fi + local sum="$(grep "$filename" "$sumfile")" + local nb_lf=${sum//[^$'\n']/} + if [[ $nb_lf -gt 0 ]]; then + printf '%s: ' "$(gettext "FAILED")" >&2 + [[ -n "$tmp" ]] && sumfile="$file.$integ.asc" + printf "$(gettext "%s invalid")" "$(get_filename "$sumfile")" >&2 + printf '\n' >&2 + [ -e "$tmp" ] && rm -f "$tmp" + return 1 + fi + expectedsum="${sum%% *}" + fi + [ -e "$tmp" ] && rm -f "$tmp" + local realsum="$(openssl dgst -${integ} "$file")" realsum="${realsum##* }" if [[ ${expectedsum,,} = "$realsum" ]]; then @@ -602,6 +649,7 @@ check_pgpsigs() { local errors=0 local statusfile=$(mktemp) local all_sources + local integ case $1 in all) @@ -613,7 +661,7 @@ check_pgpsigs() { esac for file in "${all_sources[@]}"; do file="$(get_filename "$file")" - if [[ $file != *.@(sig?(n)|asc) ]]; then + if [[ $file != *.@(sig?(n)|asc|signed-msg) ]]; then continue fi @@ -626,29 +674,33 @@ check_pgpsigs() { fi found=0 - for ext in "" gz bz2 xz lrz lzo Z; do - if sourcefile="$(get_filepath "${file%.*}${ext:+.$ext}")"; then - found=1 - break; + if [[ $file = *.signed-msg ]]; then + gpg --quiet --batch --status-file "$statusfile" --verify < "$file" 2> /dev/null + else + for ext in "" gz bz2 xz lrz lzo Z; do + if sourcefile="$(get_filepath "${file%.*}${ext:+.$ext}")"; then + found=1 + break; + fi + done + if (( ! found )); then + printf '%s\n' "$(gettext "SOURCE FILE NOT FOUND")" >&2 + errors=1 + continue fi - done - if (( ! found )); then - printf '%s\n' "$(gettext "SOURCE FILE NOT FOUND")" >&2 - errors=1 - continue - fi - case "$ext" in - gz) decompress="gzip -c -d -f" ;; - bz2) decompress="bzip2 -c -d -f" ;; - xz) decompress="xz -c -d" ;; - lrz) decompress="lrzip -q -d" ;; - lzo) decompress="lzop -c -d -q" ;; - Z) decompress="uncompress -c -f" ;; - "") decompress="cat" ;; - esac + case "$ext" in + gz) decompress="gzip -c -d -f" ;; + bz2) decompress="bzip2 -c -d -f" ;; + xz) decompress="xz -c -d" ;; + lrz) decompress="lrzip -q -d" ;; + lzo) decompress="lzop -c -d -q" ;; + Z) decompress="uncompress -c -f" ;; + "") decompress="cat" ;; + esac - $decompress < "$sourcefile" | gpg --quiet --batch --status-file "$statusfile" --verify "$file" - 2> /dev/null + $decompress < "$sourcefile" | gpg --quiet --batch --status-file "$statusfile" --verify "$file" - 2> /dev/null + fi # these variables are assigned values in parse_gpg_statusfile success=0 status= -- 2.6.2
On 06/11/15 04:48, Olivier Brunel wrote:
Verifying PGP signatures was only done by expecting them to be detached signature of the source code, which is the case on many occasions. However, some upstreams actually do things a bit differently, specifically there are two other main ways things can be done:
- a checksum file is provided, as well as a detached signature of that file. For such cases, this adds a special mode (alongside SKIP) to the sums array: FILE In that case, for filename.ext a file filename.ext.$ALGO is expected to be a standard checksum file - i.e. contain at least a line with the hash, spaces, then the filename - and the hash used will be the one from that file. Obviously this should be used when a detached signature of the file is provided, which will be treated by makepkg just as usual. An example package could be firefox.
- a checksum file is provided as a signed message. For such cases, this adds a mode SIGNED-MSG, as it expects a file filename.ext.$ALGO.signed-msg to be the signed message. Upon hash checking, it will "extract" said message from the file via gpg, then use it much like in the FILE case. And during PGP signature checking, *.signed-msg files are verified as signed messages instead of detached signatures. An example package could be harfbuzz.
Note that this doesn't change generation (--geninteg) since it can't be done automatically.
Signed-off-by: Olivier Brunel <jjk@jjacky.com>
I am going to canvas on opinions for this. My opinion is that this should not be included in makepkg because of there being no standard on the format of any of these files. You have used the output of coreutils hashing tools, but upstreams could use openssl, or anything else. If we support one, we need to support them all. Also, the packager needs to download the checksum files and their signatures to verify the source while packaging. Putting the has in the PKGBUILD prevents everyone else doing this. It also prevents upstream changing the released source (it happens...), and hash files and no-one noticing. Allan
On 11/06/15 06:12, Allan McRae wrote:
On 06/11/15 04:48, Olivier Brunel wrote:
Verifying PGP signatures was only done by expecting them to be detached signature of the source code, which is the case on many occasions. However, some upstreams actually do things a bit differently, specifically there are two other main ways things can be done:
- a checksum file is provided, as well as a detached signature of that file. For such cases, this adds a special mode (alongside SKIP) to the sums array: FILE In that case, for filename.ext a file filename.ext.$ALGO is expected to be a standard checksum file - i.e. contain at least a line with the hash, spaces, then the filename - and the hash used will be the one from that file. Obviously this should be used when a detached signature of the file is provided, which will be treated by makepkg just as usual. An example package could be firefox.
- a checksum file is provided as a signed message. For such cases, this adds a mode SIGNED-MSG, as it expects a file filename.ext.$ALGO.signed-msg to be the signed message. Upon hash checking, it will "extract" said message from the file via gpg, then use it much like in the FILE case. And during PGP signature checking, *.signed-msg files are verified as signed messages instead of detached signatures. An example package could be harfbuzz.
Note that this doesn't change generation (--geninteg) since it can't be done automatically.
Signed-off-by: Olivier Brunel <jjk@jjacky.com>
I am going to canvas on opinions for this.
My opinion is that this should not be included in makepkg because of there being no standard on the format of any of these files. You have used the output of coreutils hashing tools, but upstreams could use openssl, or anything else. If we support one, we need to support them all.
Well, I don't know... One could say the same about PGP signatures, if one is supported, all must be. Clearly it isn't the case as of now :p The format from coreutils tools is the only one I've seen used in upstreams, not that it means anything, except at least that it is a common one. As you said, there are no rules/standard for what upstream will do or format they'll use, but this one is known (in fact I believe openssl has an option to generate output in that format), and seems to be the commonly used format; and supporting it allows to support more PGP signature checking.
Also, the packager needs to download the checksum files and their signatures to verify the source while packaging. Putting the has in the PKGBUILD prevents everyone else doing this. It also prevents upstream changing the released source (it happens...), and hash files and no-one noticing.
Sorry, I don't understand what you mean by this: Putting the hash in the PKGBUILD prevents everyone else doing this Could you explain/rephrase please? I don't get what gets prevented. As for upstream changing/"repackaging" a release, if that's something you want to prevent, it can be done simply by adding another sums array. E.g. one could use the sha256sums with SIGNED-MSG/FILE to check the hash & signature from upstream, and the sha1sums (from the packager, i.e. with actual hashes) to ensure things don't silently change from upstream since original packaging.
Allan
On 06/11/15 22:59, Olivier Brunel wrote:
On 11/06/15 06:12, Allan McRae wrote:
On 06/11/15 04:48, Olivier Brunel wrote:
Verifying PGP signatures was only done by expecting them to be detached signature of the source code, which is the case on many occasions. However, some upstreams actually do things a bit differently, specifically there are two other main ways things can be done:
- a checksum file is provided, as well as a detached signature of that file. For such cases, this adds a special mode (alongside SKIP) to the sums array: FILE In that case, for filename.ext a file filename.ext.$ALGO is expected to be a standard checksum file - i.e. contain at least a line with the hash, spaces, then the filename - and the hash used will be the one from that file. Obviously this should be used when a detached signature of the file is provided, which will be treated by makepkg just as usual. An example package could be firefox.
- a checksum file is provided as a signed message. For such cases, this adds a mode SIGNED-MSG, as it expects a file filename.ext.$ALGO.signed-msg to be the signed message. Upon hash checking, it will "extract" said message from the file via gpg, then use it much like in the FILE case. And during PGP signature checking, *.signed-msg files are verified as signed messages instead of detached signatures. An example package could be harfbuzz.
Note that this doesn't change generation (--geninteg) since it can't be done automatically.
Signed-off-by: Olivier Brunel <jjk@jjacky.com>
I am going to canvas on opinions for this.
My opinion is that this should not be included in makepkg because of there being no standard on the format of any of these files. You have used the output of coreutils hashing tools, but upstreams could use openssl, or anything else. If we support one, we need to support them all.
Well, I don't know... One could say the same about PGP signatures, if one is supported, all must be. Clearly it isn't the case as of now :p
We support all detached signatures for source files.
The format from coreutils tools is the only one I've seen used in upstreams, not that it means anything, except at least that it is a common one.
As you said, there are no rules/standard for what upstream will do or format they'll use, but this one is known (in fact I believe openssl has an option to generate output in that format), and seems to be the commonly used format; and supporting it allows to support more PGP signature checking.
Also, the packager needs to download the checksum files and their signatures to verify the source while packaging. Putting the has in the PKGBUILD prevents everyone else doing this. It also prevents upstream changing the released source (it happens...), and hash files and no-one noticing.
Sorry, I don't understand what you mean by this: Putting the hash in the PKGBUILD prevents everyone else doing this
Could you explain/rephrase please? I don't get what gets prevented.
Which has goes in the PKGBUILD? I assume you would have somethimg like: source=("https://ftp.mozilla.org/pub/firefox/releases/42.0/source/firefox-42.0.source..." "firefox-42.0.source.tar.xz.sha256::https://ftp.mozilla.org/pub/firefox/releases/42.0/SHA512SUMS" "firefox-42.0.source.tar.xz.sha256.asc::https://ftp.mozilla.org/pub/firefox/releases/42.0/SHA512SUMS.asc") sha256sums=("FILE" "SKIP" "SKIP") With that, upstream can change the file on the server and you will not notice. If you don't use SKIP for the sha256 file, you need to update the PKGBUILD every time upstream ads a new release to the file.
As for upstream changing/"repackaging" a release, if that's something you want to prevent, it can be done simply by adding another sums array. E.g. one could use the sha256sums with SIGNED-MSG/FILE to check the hash & signature from upstream, and the sha1sums (from the packager, i.e. with actual hashes) to ensure things don't silently change from upstream since original packaging.
If you are putting other sums in the array, why not just put the sums from the signed file into the PKGBUILD after checking it?
On 11/07/15 06:50, Allan McRae wrote:
On 06/11/15 22:59, Olivier Brunel wrote:
On 11/06/15 06:12, Allan McRae wrote:
On 06/11/15 04:48, Olivier Brunel wrote:
Verifying PGP signatures was only done by expecting them to be detached signature of the source code, which is the case on many occasions. However, some upstreams actually do things a bit differently, specifically there are two other main ways things can be done:
- a checksum file is provided, as well as a detached signature of that file. For such cases, this adds a special mode (alongside SKIP) to the sums array: FILE In that case, for filename.ext a file filename.ext.$ALGO is expected to be a standard checksum file - i.e. contain at least a line with the hash, spaces, then the filename - and the hash used will be the one from that file. Obviously this should be used when a detached signature of the file is provided, which will be treated by makepkg just as usual. An example package could be firefox.
- a checksum file is provided as a signed message. For such cases, this adds a mode SIGNED-MSG, as it expects a file filename.ext.$ALGO.signed-msg to be the signed message. Upon hash checking, it will "extract" said message from the file via gpg, then use it much like in the FILE case. And during PGP signature checking, *.signed-msg files are verified as signed messages instead of detached signatures. An example package could be harfbuzz.
Note that this doesn't change generation (--geninteg) since it can't be done automatically.
Signed-off-by: Olivier Brunel <jjk@jjacky.com>
I am going to canvas on opinions for this.
My opinion is that this should not be included in makepkg because of there being no standard on the format of any of these files. You have used the output of coreutils hashing tools, but upstreams could use openssl, or anything else. If we support one, we need to support them all.
Well, I don't know... One could say the same about PGP signatures, if one is supported, all must be. Clearly it isn't the case as of now :p
We support all detached signatures for source files.
The format from coreutils tools is the only one I've seen used in upstreams, not that it means anything, except at least that it is a common one.
As you said, there are no rules/standard for what upstream will do or format they'll use, but this one is known (in fact I believe openssl has an option to generate output in that format), and seems to be the commonly used format; and supporting it allows to support more PGP signature checking.
Also, the packager needs to download the checksum files and their signatures to verify the source while packaging. Putting the has in the PKGBUILD prevents everyone else doing this. It also prevents upstream changing the released source (it happens...), and hash files and no-one noticing.
Sorry, I don't understand what you mean by this: Putting the hash in the PKGBUILD prevents everyone else doing this
Could you explain/rephrase please? I don't get what gets prevented.
Which has goes in the PKGBUILD? I assume you would have somethimg like:
source=("https://ftp.mozilla.org/pub/firefox/releases/42.0/source/firefox-42.0.source..."
"firefox-42.0.source.tar.xz.sha256::https://ftp.mozilla.org/pub/firefox/releases/42.0/SHA512SUMS" "firefox-42.0.source.tar.xz.sha256.asc::https://ftp.mozilla.org/pub/firefox/releases/42.0/SHA512SUMS.asc")
sha256sums=("FILE" "SKIP" "SKIP")
(The filenames & array should read sha512, not sha256, but yes.)
With that, upstream can change the file on the server and you will not notice.
If you don't use SKIP for the sha256 file, you need to update the PKGBUILD every time upstream ads a new release to the file.
Right; Okay so I think in your original mail, that last paragraph ("Also, ...") was meant to be informative, right? Just stating how packaging should be done (i.e. not use ('FILE' 'SKIP' 'SKIP') but put the hash of the checksum file in, for reasons stated), whereas I thought you meant there was something that, with the use of those modes, wasn't possible (anymore). I believe that was the source of my misunderstanding. So, yes, the hash of the checksum file should go in, as explained, and indeed you don't even need another sum array.
As for upstream changing/"repackaging" a release, if that's something you want to prevent, it can be done simply by adding another sums array. E.g. one could use the sha256sums with SIGNED-MSG/FILE to check the hash & signature from upstream, and the sha1sums (from the packager, i.e. with actual hashes) to ensure things don't silently change from upstream since original packaging.
If you are putting other sums in the array, why not just put the sums from the signed file into the PKGBUILD after checking it?
participants (2)
-
Allan McRae
-
Olivier Brunel