Le 16/12/2016 à 00:13, Allan McRae a écrit :

On 16/12/16 08:29, Bruno Pagani wrote:

...

Hi there,

This is probably some sort of feature request or maybe more general asking.

I have a case where sha*sums are provided in a .dsc signed file (bs1770gain, for which *sane* upstream is Debian: https://packages.debian.org/source/sid/bs1770gain). Apparently, makepkg only supports verifying file with detached signature. Is there a specific reason for that (like this use case is really tiny — I have no actual idea about this) or is it just because it was never implemented?

The format of signed checksum files varies a lot. I don't want to attempt to autodetect each one as that will create a future nightmare.

Also, what is gained vs putting the checksums into the PKGBUILD?

A

Easy verification of source for other people building the package (here from AUR, but could be from ABS). Or maybe I misunderstood the point of having PGP verification in makepkg? The idea is to be able to just put the .dsc file in the sources array and extract the checksum from it in the sha*sums array (here I just write "$(grep -A1 Sha256 ${pkgname}_${pkgver}-1.dsc | tail -n 1 | cut -d ' ' -f 2)" in there), while having makepkg verify the .dsc file. Anyway, I understand your point, I never thought the format could be something else than a PGP signed message with or without detached signature. Or maybe you were thinking of extracting the sha*sums (which is then probably linked to my first point — my possible misunderstanding of PGP verification in makepkg)? Thanks, Bruno

Le 16/12/2016 à 01:00, Allan McRae a écrit :

On 16/12/16 09:23, Bruno Pagani wrote:

...

Le 16/12/2016 à 00:13, Allan McRae a écrit :

...

On 16/12/16 08:29, Bruno Pagani wrote:

...

Hi there,

This is probably some sort of feature request or maybe more general asking.

I have a case where sha*sums are provided in a .dsc signed file (bs1770gain, for which *sane* upstream is Debian: https://packages.debian.org/source/sid/bs1770gain). Apparently, makepkg only supports verifying file with detached signature. Is there a specific reason for that (like this use case is really tiny — I have no actual idea about this) or is it just because it was never implemented?

The format of signed checksum files varies a lot. I don't want to attempt to autodetect each one as that will create a future nightmare.

Also, what is gained vs putting the checksums into the PKGBUILD?

A Easy verification of source for other people building the package (here from AUR, but could be from ABS). Or maybe I misunderstood the point of having PGP verification in makepkg?

PGP verification in makepkg is to verify PGP signatures for sources. Here, we just get checksums that are signed. So it only used the checksum to verify the source. Just putting the checksum in the PKGBUILD provides the same (assuming you trust the PKGBUILD...)

On AUR, trusting the PKGBUILD is not a good idea. ;) But I think that even in the main repos it does matter, because from one glance at the PKGBUILD I can see that the maintainer is aware of the existence of this signed file and make use of it. And else what is the difference then with this case: sources are signed, the maintainer has verified the signature and then just added the (calculated/provided) sha*sum to the PKGBUILD?

...

The idea is to be able to just put the .dsc file in the sources array and extract the checksum from it in the sha*sums array (here I just write "$(grep -A1 Sha256 ${pkgname}_${pkgver}-1.dsc | tail -n 1 | cut -d ' ' -f 2)" in there), while having makepkg verify the .dsc file. The Debian format is well defined, but other upstream sources provide similar signed checksum files that have varying formats.

But does checking them with `gpg --verify ` work? Because that’s all my point is about. I don’t want makepkg to extract the sums from the file, that would be done in the PKGBUILD. However, if I could get it to verify that file that would be nice. ;) Anyway, as you can see in my answer to Eli, I might not be needing that anymore actually, but I’m still wondering whether makepkg should verify with GPG any files in .dsc provided in the source array like it does for .asc/.sig files. Once again, not asking to do anything clever with the actual content of that file, just verifying its sig. My point is really about verifying attached sigs like it’s done for detached ones. Thanks, Bruno

Le 16/12/2016 à 05:57, Eli Schwartz a écrit :

On 12/15/2016 06:23 PM, Bruno Pagani wrote:

...

Le 16/12/2016 à 00:13, Allan McRae a écrit :

...

On 16/12/16 08:29, Bruno Pagani wrote:

...

Hi there,

This is probably some sort of feature request or maybe more general asking.

I have a case where sha*sums are provided in a .dsc signed file (bs1770gain, for which *sane* upstream is Debian: https://packages.debian.org/source/sid/bs1770gain). Apparently, makepkg only supports verifying file with detached signature. Is there a specific reason for that (like this use case is really tiny — I have no actual idea about this) or is it just because it was never implemented?

The format of signed checksum files varies a lot. I don't want to attempt to autodetect each one as that will create a future nightmare.

Also, what is gained vs putting the checksums into the PKGBUILD?

A Easy verification of source for other people building the package (here from AUR, but could be from ABS). Or maybe I misunderstood the point of having PGP verification in makepkg? PGP verification proves that upstream signed the sources. Debian .dsc files prove nothing other than that Debian signed that download.

You’re right about this, but in this particular case Debian does import and check the code in their own tree on each release, so their signature has a meaning to me. And also, they used to modify it (the source code), but thanks to your below question I have checked that again and it’s no longer the case. So in some way it’s like Debian signed some sort of code audit, or regarding the second point like they’ve forked the project and signed they own releases.

What exactly qualifies Debian as the "sane upstream" anyway?

Well, at some point bs1770gain build system was downloading needed libs from around the internet at build time rather than linking against system ones, but that has been fixed upstream. I’m probably going to switch to upstream unsigned tarball, since the second point (“forked project”) doesn’t apply anymore, though the first one is still valid IMHO. If you have any opinion on this, please share it. ;)

What does authenticating Debian's checksums get us, that we couldn't have gotten out of verifying the AUR maintainer's checksums?

Not sure to understand that one: what’s the point of PGP at all in makepkg then? Like I’ve said in my other emails responding to Allan, how is this different from a signed source verified by the maintainer while having just the checksum pasted in the PKGBUILD? Or if it’s just because of Debian vs real upstream again, see my point above and below.

Note that Allan has already vetoed the idea of upgrading the default checksum type in makepkg, on the grounds that it doesn't really prove anything, and nothing other than actual checksums or preferably PGP signatures from the code author will prove *anything*.

And I totally agree with that. The ambiguity here is around “code author”, since in that case I consider Debian to fulfil this role, but I definitively would like to hear your opinion on this now that I (hope to) have enlightened some points.

So I don't see why yet more unproven checksums will be any different, especially if it requires brand-new handling in makepkg specifically for the purpose.

Agree again, but my point is not about this package in particular, it’s in the event of upstream providing sha*sums in a signed file as opposed to a detached signature. Not that currently makepkg supports verifying a sha*sums with detached signatures (maybe not by design but as consequence of source signature verification). Bruno

On Fri, 16 Dec 2016 14:52:20 -0500 Eli Schwartz wrote: (...)

Well, Firefox upstream for one supplies sha512sums in a signed file.[1] So this could in theory be used.

The problem is that you can copy the checksums into the PKGBUILD and PGP-verify the checksum file, but unless you seriously reorganize makepkg's verification logic you cannot download the checksum file, PGP-verify it and *then* check the other files based on the checksum file. And I don't think anyone else strongly cares about doing that, but maybe if you provided a patch it would be accepted?

Well, for the record there is a patch[1] for doing just that (and a bit more) actually. Because indeed a few upstreams do not provide signatures of the source code directly, but either detached sig of a checksum file, or checksums as a signed message. The patch in question handles both cases. And as it happens, it will work with firefox upstream, amongst others. (Though not with the .dsc files from Debian mentionned in this thread.) Cheers, [1] https://lists.archlinux.org/pipermail/pacman-dev/2015-November/020564.html

On 12/16/2016 04:04 PM, Eli Schwartz wrote:

Hmm, I had forgotten that.

Or never saw it, actually. I am not sure I was checking the list that long ago, and may be misremembering remembering. :( -- Eli Schwartz

Le 16/12/2016 à 22:24, Olivier Brunel a écrit :

On Fri, 16 Dec 2016 16:04:06 -0500 Eli Schwartz wrote:

...

...

Well, for the record there is a patch[1] for doing just that (and a bit more) actually. Because indeed a few upstreams do not provide signatures of the source code directly, but either detached sig of a checksum file, or checksums as a signed message. The patch in question handles both cases.

And as it happens, it will work with firefox upstream, amongst others. (Though not with the .dsc files from Debian mentionned in this thread.)

Cheers,

[1] https://lists.archlinux.org/pipermail/pacman-dev/2015-November/020564.html Hmm, I had forgotten that. I see that Allan objected to that on the grounds that upstream could re-release the sums e.g. after adding a new artifact to the hundred or so in the Firefox file. So you would either have spurious failures, or be unable to detect re-releases. Not exactly, as long as you put the hash of the file in the PKGBUILD, any change from upstream would be cought. I believe what Allan pointed out was that using SKIP for the file could lead to such things, but

On 12/16/2016 03:40 PM, Olivier Brunel wrote: that would be a packaging rule to follow to ensure things don't happen.

I totally agree with this. :) Quite funnily, this is why I thought the feature I would like makepkg to have was easy to have, because having already downloaded the signed file to add its sha*sum to the corresponding array, it allowed makepkg to correctly parse it with whatever I had put in the sha*sum array, while this doesn’t work if the file isn’t already downloaded. Bruno

Le 16/12/2016 à 20:52, Eli Schwartz a écrit :

...

You’re right about this, but in this particular case Debian does import and check the code in their own tree on each release, so their signature has a meaning to me. And also, they used to modify it (the source code), but thanks to your below question I have checked that again and it’s no longer the case. So in some way it’s like Debian signed some sort of code audit, or regarding the second point like they’ve forked the project and signed they own releases. Not really true in the general case. :p There are several somewhat famous Debian bugs regarding software they shipped that did what was to

On 12/16/2016 08:37 AM, Bruno Pagani wrote: them very offensive things, and the maintainer got thoroughly scolded for having failed to catch it.

Indeed, never thought of “Debian OpenSSL keys” from that perspective. :p

Point being, we are all human. Supposedly, Arch/AUR maintainers check the code they use too. Debian are not some magical group of exalted beings with higher standards of correctness that can be depended on. I imagine careless AUR maintainers are no more common than careless Debian maintainers -- or vice versa! ;)

That’s right in general indeed, and starting to do case by case is probably not a good idea. ;)

...

Well, at some point bs1770gain build system was downloading needed libs from around the internet at build time rather than linking against system ones, but that has been fixed upstream. I’m probably going to switch to upstream unsigned tarball, since the second point (“forked project”) doesn’t apply anymore, though the first one is still valid IMHO. If you have any opinion on this, please share it. ;) Well then, they are doing a better job packaging this than they did in the openssl fiasco. ;)

Definitively. :)

Joking aside, that still doesn't make them an upstream, ever. I would still recommend using official upstream-sanctioned sources

I’ll switch bs1770gain to that then. But I have the impression you’re seing that even before I should have used upstream source despite their ugly/shadowy build system, is that the case? Maybe while carrying Debian patch to build correctly aside?

...

...

What does authenticating Debian's checksums get us, that we couldn't have gotten out of verifying the AUR maintainer's checksums? Not sure to understand that one: what’s the point of PGP at all in makepkg then? Like I’ve said in my other emails responding to Allan, how is this different from a signed source verified by the maintainer while having just the checksum pasted in the PKGBUILD? PGP in makepkg does whatever you tell it to, which unlike checksums is *capable* of supplying authority in addition to integrity.

Yes, but signed sums are just one more step away from this setup.

...

Or if it’s just because of Debian vs real upstream again, see my point above and below. But yes, it is "Debian vs. real upstream", and I am not yet convinced.

And you’re right not to be, since you stand with the valid opinion here. ;)

...

And I totally agree with that. The ambiguity here is around “code author”, since in that case I consider Debian to fulfil this role, but I definitively would like to hear your opinion on this now that I (hope to) have enlightened some points. They are only an author for their modifications, and as for the rest they are a code *auditor*. But so are Arch/AUR maintainers.

And see how that works out, in both projects, at scale. :(

OK, fair point, I think it answers my above questions by yes and yes. :)

...

Agree again, but my point is not about this package in particular, it’s in the event of upstream providing sha*sums in a signed file as opposed to a detached signature. Not that currently makepkg supports verifying a sha*sums with detached signatures (maybe not by design but as consequence of source signature verification). Well, Firefox upstream for one supplies sha512sums in a signed file.[1] So this could in theory be used.

The problem is that you can copy the checksums into the PKGBUILD and PGP-verify the checksum file, but unless you seriously reorganize makepkg's verification logic you cannot download the checksum file, PGP-verify it and *then* check the other files based on the checksum file. And I don't think anyone else strongly cares about doing that, but maybe if you provided a patch it would be accepted?

Interesting, ever thought it worked, but apparently it was because the file was already downloaded in my workdir (since I needed the sha*sum of the sha*sums file — see why in Olivier Brunel answer). I had this in my bs1770gain PKGBUILD: source=("http://http.debian.net/debian/pool/main/b/${pkgname}/${pkgname}_${pkgver}"{.orig.tar.gz,-1.dsc}) sha256sums=("$(grep -A1 Sha256 ${pkgname}_${pkgver}-1.dsc | tail -n 1 | cut -d ' ' -f 2)" '4011c60f797c55ddf1eb0ce6ad981bc48a70a96bc7d7b3152ddb740cc2b46bbb') And it worked like a charm, but after deleting the .dsc file it indeed failed. Just relaunching makepkg was enough to make it success, but that’s not a sane workflow. I’m not familiar with makepkg code, but would it really be that hard to allow such things to work? I’m not sure why makepkg can’t see the downloaded file the first time but can the second, is that because the commands in sha256sums are executed before downloading?

The other problem is formats for checksum files. ;) Theoretically, the reusable output of sha*sum is a standard, but in practice some use the BSD-style type, and as witnessed in the Firefox checksum manifest, there may be many files listed in one checksum file using complex paths, so that would have to be filtered.

Or just allow the above setup and let the parsing of the checksum file at maintainer responsibility?

As for attached signatures, that is hardly a Debian-specific idea, but I confess I don't usually see file resources with attached signatures floating around. The thing about them is that you need to have gpg in order to transform the file into something usable -- that's why the detached signature exists.

Yes indeed, file resources with attached signature are not likely to exist. But this is really about signed checksums.

So I think it is, in practice, more or less limited to inline signing of *messages*, and makepkg doesn't usually need to worry about those. But in theory we could fix that and teach makepkg how to recognize sources with builtin signatures. (Actually the case with git, and I had to special-case it to add signed commit/tag support.)

Yes indeed, it’s all about inline signing of messages, those messages being sha*sums of the resource file. ;) Bruno

Le 18/12/2016 à 02:08, Eli Schwartz a écrit :

On 12/16/2016 05:08 PM, Bruno Pagani wrote:

...

Le 16/12/2016 à 20:52, Eli Schwartz a écrit :

...

Not really true in the general case. :p There are several somewhat famous Debian bugs regarding software they shipped that did what was to them very offensive things, and the maintainer got thoroughly scolded for having failed to catch it. Indeed, never thought of “Debian OpenSSL keys” from that perspective. :p Um, actually, I've never thought about it from that perspective either, I think...

The openssl fiasco wasn't really a case of the source code doing things that violate Debian standards (and hence is offensive *to Debian* specifically). It was *because* of their tampering that it did the wrong thing, but not necessarily an "offensive" thing.

Yes, but I see it more like “Debian tampering of upstream source” in general, whether that’s offensive for them or not. ;) My point was: Debian is not necessarily more trustworthy than upstream.

I was actually thinking about xscreensaver.[1]

OK, that is far too long to read (only read a quarter of it, with “nice” exchanges between users, Debian maintainers and upstream dev), would you mind making me a summary (or pointing to a specific comment in the thread doing so if there is one)?

...

I’ll switch bs1770gain to that then. But I have the impression you’re seing that even before I should have used upstream source despite their ugly/shadowy build system, is that the case? Maybe while carrying Debian patch to build correctly aside? Correct. I agree that things should build against system libraries (except in cases like FS#43382 in which case the appropriate response is to work with upstream to make the software work properly with system libraries).

Patching the build system is okay (but dirty) in that case, as long as upstream is notified in due course (so it can stop being dirty).

However, I strongly believe that should only be done as patches applied to the unmodified and verifiable upstream source code. If the patches are from Debian, I am not suggesting we hate on Debian and reject it. ;) I am simply saying, use their patches and upstream's release tarball, rather than using Debian's modified tarball.

Thanks for this, will remember it. And FS#43382 was (or even is, since it is apparently still living) quite interesting too. ;)

...

...

...

...

What does authenticating Debian's checksums get us, that we couldn't have gotten out of verifying the AUR maintainer's checksums? Not sure to understand that one: what’s the point of PGP at all in makepkg then? Like I’ve said in my other emails responding to Allan, how is this different from a signed source verified by the maintainer while having just the checksum pasted in the PKGBUILD? PGP in makepkg does whatever you tell it to, which unlike checksums is *capable* of supplying authority in addition to integrity. Yes, but signed sums are just one more step away from this setup. True, and signed sums generated by a "real upstream" is something I wouldn't mind having. But authenticating Debian's checksums is no better or worse than having an AUR maintainer ship a signature generated from the AUR maintainer's *own* GPG key!

I disapprove of that in the AUR, so I won't hold us to a higher standard than I hold Debian. :)

Definitively agreed, that’s why this topic moved to verifying upstream .dsc signed sums. ;)

...

I’m not familiar with makepkg code, but would it really be that hard to allow such things to work? I’m not sure why makepkg can’t see the downloaded file the first time but can the second, is that because the commands in sha256sums are executed before downloading? Correct. Towards the beginning of makepkg's runtime, it sources the PKGBUILD and at that time the grep magic occurs, because a PKGBUILD can contain any kind of bash hacks.

However, ultimately the job of a PKGBUILD is to define certain variables and functions (at the time it is sourced), which makepkg will then use.

Later on, makepkg does its "download all the sources" thing, looping through the source array and calling appropriate download agents to retrieve those sources. And later on yet, makepkg loops through the sources again and verifies them via the *sums array(s) and PGP signatures if available

What you can do since you are after all relying on hacks ;) is to pre-download the dsc. You can always depend on curl to exist since pacman relies on it, and essentially it can be treated like a local source.

That looks fine for now. And it’s probably the most only thing we can do in a reasonable time frame, since working around this in makepkg looks definitively more hard than I thought. As a matter of fact, I’ve been doing that in firefox-nightly-fr[1] for quite some time (this PKGBUILD could need some simplification now that I’ve learned much more things, will do this in my current PKGBUILD enhancement tour). But in the case of upstream signed .dsc sums, this still require some supplemental support on makepkg side. Do you think this could be achievable easily? Even if to be honest, I’m not sure of any upstream providing such signed sums, and that would be definitively OK then to wait for actual cases to be reported before being implemented, so for now an answer on whether this would be accepted if needed for this use case would be sufficient. But that’s probably more for Allan to answer this than you. ;)

...

...

The other problem is formats for checksum files. ;) Theoretically, the reusable output of sha*sum is a standard, but in practice some use the BSD-style type, and as witnessed in the Firefox checksum manifest, there may be many files listed in one checksum file using complex paths, so that would have to be filtered. Or just allow the above setup and let the parsing of the checksum file at maintainer responsibility?

...

As for attached signatures, that is hardly a Debian-specific idea, but I confess I don't usually see file resources with attached signatures floating around. The thing about them is that you need to have gpg in order to transform the file into something usable -- that's why the detached signature exists. Yes indeed, file resources with attached signature are not likely to exist. But this is really about signed checksums.

...

So I think it is, in practice, more or less limited to inline signing of *messages*, and makepkg doesn't usually need to worry about those. But in theory we could fix that and teach makepkg how to recognize sources with builtin signatures. (Actually the case with git, and I had to special-case it to add signed commit/tag support.) Yes indeed, it’s all about inline signing of messages, those messages being sha*sums of the resource file. ;) All this ^^ is really about "what is likely enough that we should build special handling into libmakepkg, rather than requiring each maintainer to reinvent the wheel/do more busywork".

Yes, and don’t you think that allowing easy verification of .dsc file would be nice (i.e. without having to code it in a prepare{} function or the like), even if the parsing and downloading of the sum file is still to be handled by the maintainer in the PKGBUILD? Again, just wanting to know whether that would be accepted in case of need, but since I, at least, have none at the moment, there is no urge in implementing this unless someone actually comes in with a lot of reported cases. Thanks for all your enlightenment and contributions to this discussion, Bruno [1]: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=firefox-nightly-fr

On 12/21/2016 11:38 AM, Bruno Pagani wrote:

OK, that is far too long to read (only read a quarter of it, with “nice” exchanges between users, Debian maintainers and upstream dev), would you mind making me a summary (or pointing to a specific comment in the thread doing so if there is one)?

Upstream considers his software to be security-critical (fair enough, it is a lockscreen), Debian doesn't like updating things, so xscreensaver contains a vaguely threatening out-of-date warning. At a certain point, people started getting warnings that xscreensaver was out of date and "your distro packager is pathetic and horrible, don't ever use _____ (mainly Debian) because they don't ever update my software, blah blah blah" leading to worried bugreports. So they "had" to patch out the warning, and in the meantime the usual "people on the internet" took the opportunity to comment on the author and his anti-linux tendencies. The Debian maintainer also got egg on his face for not having actually looked at the source code diffs, considering how noticeable the warning was. (There was a rather lengthy comment imploring distro maintainers to be polite and leave the warning in, on account of the author strongly dislikes bugreports from people who have been mislead by Debian into running ancient versions that have long-since-fixed bugs. It really *should* have been noticed.) The maintainer was human. :)

Thanks for this, will remember it. And FS#43382 was (or even is, since it is apparently still living) quite interesting too. ;)

Incidentally, a point in Debian's favor -- they seem to be aware of how html5lib was forked, and use the bundled fork so the app works; Arch only conceded the point when the API changed and calibre's fork didn't.

That looks fine for now. And it’s probably the most only thing we can do in a reasonable time frame, since working around this in makepkg looks definitively more hard than I thought. As a matter of fact, I’ve been doing that in firefox-nightly-fr[1] for quite some time (this PKGBUILD could need some simplification now that I’ve learned much more things, will do this in my current PKGBUILD enhancement tour).

But in the case of upstream signed .dsc sums, this still require some supplemental support on makepkg side. Do you think this could be achievable easily? Even if to be honest, I’m not sure of any upstream providing such signed sums, and that would be definitively OK then to wait for actual cases to be reported before being implemented, so for now an answer on whether this would be accepted if needed for this use case would be sufficient. But that’s probably more for Allan to answer this than you. ;)

I still don't see the problem. You can *always* do whatever makepkg isn't doing yourself, as long as the .dsc source file is downloaded before *sums=() is sourced. Yes, it is still an ugly hack and not really supported, but the hack should work...

Yes, and don’t you think that allowing easy verification of .dsc file would be nice (i.e. without having to code it in a prepare{} function or the like), even if the parsing and downloading of the sum file is still to be handled by the maintainer in the PKGBUILD? Again, just wanting to know whether that would be accepted in case of need, but since I, at least, have none at the moment, there is no urge in implementing this unless someone actually comes in with a lot of reported cases.

If makepkg were going to support it at all, it wouldn't be halfway. End that train of thought. -- Eli Schwartz