After posting a message to this list earlier today, I immediately received nearly a dozen DKIM fail messages, all being sent by the "OpenDMARC Filter" at various domains, and all saying that the DKIM fail reason was "signature verification failed".
I raised this issue with support at my hosting company, who pointed out that a) my domain's DKIM records are set up correctly, and b) one of the headers on my email that got sent to the list indicated that Arch's mailman actually did successfully verify my domain's DKIM record when it received my email, before forwarding it on.
Anyone know why these fail messages might be happening? Is this being caused by a misconfiguration in Arch's mailman installation? Or is this a misconfiguration of the individual OpenDMARC software installations at each of those domains?
Thanks,
DR
On Sunday, 30 October 2022 at 17:58 (-0400), David Rosenstrauch wrote:
After posting a message to this list earlier today, I immediately received nearly a dozen DKIM fail messages, all being sent by the "OpenDMARC Filter" at various domains, and all saying that the DKIM fail reason was "signature verification failed".
[...]
Anyone know why these fail messages might be happening? Is this being caused by a misconfiguration in Arch's mailman installation? Or is this a misconfiguration of the individual OpenDMARC software installations at each of those domains?
FWIW, my OpenDKIM with default settings flagged your earlier email with a DKIM fail, but passed this one. The failure mechanism on the first email was "signature verification failed". I'm no DKIM expert, but perhaps there was a DNS resolution problem at that time and the key was inaccessible?
Relevant part of received headers follows:
From your earlier email:
Authentication-Results: mail.kent-dobias.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=darose.net header.i=@darose.net header.a=rsa-sha256 header.s=dreamhost header.b=UaDsk2dh Authentication-Results: mail.kent-dobias.com; dmarc=fail (p=none dis=none) header.from=darose.net Authentication-Results: mail.kent-dobias.com; spf=pass smtp.mailfrom=lists.archlinux.org Authentication-Results: lists.archlinux.org; dkim=pass header.d=darose.net header.s=dreamhost header.b=UaDsk2dh; dmarc=pass (policy=none) header.from=darose.net; spf=pass (lists.archlinux.org: domain of darose@darose.net designates 23.83.214.25 as permitted sender) smtp.mailfrom=darose@darose.net; arc=pass ("mailchannels.net:s=arc-2022:i=1")
From this email:
Authentication-Results: mail.kent-dobias.com; dkim=pass (2048-bit key; unprotected) header.d=darose.net header.i=@darose.net header.a=rsa-sha256 header.s=dreamhost header.b=JyPU2yJv
Authentication-Results: mail.kent-dobias.com; dmarc=pass (p=none dis=none)
header.from=darose.netAuthentication-Results: mail.kent-dobias.com; spf=pass smtp.mailfrom=lists.archlinux.org Authentication-Results: lists.archlinux.org; dkim=pass header.d=darose.net header.s=dreamhost header.b=JyPU2yJv; dmarc=pass (policy=none) header.from=darose.net; spf=pass (lists.archlinux.org: domain of darose@darose.net designates 23.83.212.19 as permitted sender) smtp.mailfrom=darose@darose.net; arc=pass ("mailchannels.net:s=arc-2022:i=1")
Jaron
Hi David / Jaron
One thought - typically dmarc gets dkim validation from a separate milter dmarc.
And many folks still run opendkim which is unmaintained. Versions prior to 2.11.0.Beta2 (which arch kindly offers) are definitely broken. I don't even trust that one.
Jaron do you run opendkim by chance?
dkimpy-milter seems to work much better than opendkim, though of course you cannot help what other people's servers run.
I see your dkim is relaxed/relaxed which should help keep your mail flowing as well.
Just a thought.
gene
On 10/30/22 6:13 PM, Jaron Kent-Dobias wrote:
On Sunday, 30 October 2022 at 17:58 (-0400), David Rosenstrauch wrote:
Anyone know why these fail messages might be happening?
FWIW, my OpenDKIM with default settings flagged your earlier email with a DKIM fail, but passed this one. The failure mechanism on the first email was "signature verification failed". I'm no DKIM expert, but perhaps there was a DNS resolution problem at that time and the key was inaccessible?
Hmmm .... that's really weird that the 1st msg failed but the 2nd passed.
And this part in particular makes no sense to me:
Relevant part of received headers follows:
From your earlier email:
Authentication-Results: mail.kent-dobias.com; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=darose.net header.i=@darose.net header.a=rsa-sha256 header.s=dreamhost header.b=UaDsk2dh Authentication-Results: mail.kent-dobias.com; dmarc=fail (p=none dis=none) header.from=darose.net Authentication-Results: mail.kent-dobias.com; spf=pass smtp.mailfrom=lists.archlinux.org Authentication-Results: lists.archlinux.org; dkim=pass header.d=darose.net header.s=dreamhost header.b=UaDsk2dh; dmarc=pass (policy=none) header.from=darose.net; spf=pass (lists.archlinux.org: domain of darose@darose.net designates 23.83.214.25 as permitted sender) smtp.mailfrom=darose@darose.net; arc=pass ("mailchannels.net:s=arc-2022:i=1")
So on the 1st msg, Arch's mailman was able to verify my DKIM record, but your OpenDKIM was not. And presumably these were only a few minutes apart from each other. (Possibly even seconds apart.)
I guess it's working now, so all's well that ends well? But still really weird.
Thanks,
DR
On 10/30/22 6:27 PM, Genes Lists wrote:
Hi David / Jaron
One thought - typically dmarc gets dkim validation from a separate milter dmarc.
And many folks still run opendkim which is unmaintained. Versions prior to 2.11.0.Beta2 (which arch kindly offers) are definitely broken. I don't even trust that one.
Jaron do you run opendkim by chance?
dkimpy-milter seems to work much better than opendkim, though of course you cannot help what other people's servers run.
I see your dkim is relaxed/relaxed which should help keep your mail flowing as well.
Just a thought.
gene
Hmmm ... looking again, it looks like every single one of those bounce msgs came from "User-Agent: OpenDMARC-Filter/1.3.2". Perhaps there's some issue in that version of that software?
That said, Jason said that his "OpenDKIM" got triggered, so maybe not.
I just got another 5 bounce messages to another one of my list emails.
This is a mystery to me. But whatever. My messages seem to be getting through to the list. And this seems to somehow be limited to me / my domain / my hosting company, and not a problem for everybody else. So probably best to just move on.
Thanks,
DR
For what it's worth, I get the same from time to time.
I've blamed my setup and/or OpenDKIM up until this point. And I will try to evaluate the recommendations from Gene as well as correlate my error messages with the headers given here.
//Anton
On 10/30/22 23:37, David Rosenstrauch wrote:
I just got another 5 bounce messages to another one of my list emails.
DR
On Sunday, 30 October 2022 at 18:30 (-0400), David Rosenstrauch wrote:
I guess it's working now, so all's well that ends well? But still really weird.
Maybe not - your latest email also failed DKIM for me. Regarding Genes' email, python-dkim's 'dkimverify' utility also registers a fail.
One thing I just noticed is that both of your emails that failed have "Content-Transfer-Encoding: base64", while the one that succeeded has "Content-Transfer-Encoding: 7bit". When I 'cat' the message in my maildir, the result of the first is base64 encoded gibberish, while the latter gives human-readable text. Perhaps something about the content transfer encoding is causing the problem?
Jaron
On Sunday, 30 October 2022 at 23:45 (+0100), Jaron Kent-Dobias wrote:
Perhaps something about the content transfer encoding is causing the problem?
After a quick google, this appears to be a known and old problem: https://stbuehler.de/blog/article/2011/05/19/dkim_fails_at_content-transfer-...
Jaron
On 10/30/22 6:48 PM, Jaron Kent-Dobias wrote:
On Sunday, 30 October 2022 at 23:45 (+0100), Jaron Kent-Dobias wrote:
Perhaps something about the content transfer encoding is causing the problem?
After a quick google, this appears to be a known and old problem: https://stbuehler.de/blog/article/2011/05/19/dkim_fails_at_content-transfer-...
Jaron
Good find - thanks!
I'm not even sure where/how that encoding is being set. Must be something that Thunderbird is doing. I'll do some digging into it.
Thanks again,
DR
On Sunday, 30 October 2022 at 18:52 (-0400), David Rosenstrauch wrote:
On 10/30/22 6:48 PM, Jaron Kent-Dobias wrote:
On Sunday, 30 October 2022 at 23:45 (+0100), Jaron Kent-Dobias wrote:
Perhaps something about the content transfer encoding is causing the problem?
After a quick google, this appears to be a known and old problem: https://stbuehler.de/blog/article/2011/05/19/dkim_fails_at_content-transfer-...
Good find - thanks!
I'm not even sure where/how that encoding is being set. Must be something that Thunderbird is doing. I'll do some digging into it.
Doing some testing of my own (via mutt), it appears to be something the client does automatically when non-ascii characters are included in the mail.
I just checked, and OpenDKIM correctly verifies signatures for base8 encoded emails sent directly to my server. Consider this email as a test for what the Arch Liunx lists do: α, β, γ, …
Jaron
On Sunday, 30 October 2022 at 23:56 (+0100), Jaron Kent-Dobias wrote:
Consider this email as a test for what the Arch Liunx lists do: α, β, γ, …
Confirmation: when Arch Linux forwards a base8 encoded email to the list, it mangles the DKIM. It does appear to be an Arch problem!
Now to deal with the litany of DMARC reports I just received...
Jaron
On Sunday, 30 October 2022 at 23:57 (+0100), Jaron Kent-Dobias wrote:
Confirmation: when Arch Linux forwards a base8 encoded email to the list, it mangles the DKIM. It does appear to be an Arch problem!
One last email: what the lists are specifically doing is rewriting 8bit encoded emails in a base64 encoding.
From the email in my sent folder:
Content-Transfer-Encoding: 8bit
From the email I received from the list:
Content-Transfer-Encoding: base64
Rewriting the body in a new encoding breaks DKIM.
Jaron
On 10/30/22 19:04, Jaron Kent-Dobias wrote:
On Sunday, 30 October 2022 at 23:57 (+0100), Jaron Kent-Dobias wrote:
Confirmation: when Arch Linux forwards a base8 encoded email to the list, it mangles the DKIM. It does appear to be an Arch problem!
Excellent debugging - congrats for finding the source of the problem.
gene
In the chain sender => list.archlinux.org => list members, I suspect it's Mailman (on the Arch list server) that does not support 8BITMIME; since OpenDKIM on that server succesfully verified the DKIM signature, the mail arrived there as 8bit, but is being distributed to list members as 7-bit.
Could the Arch listserver set "disable_mime_output_conversion=yes" in its master.cf at the point it is handing over mail to Mailman? (not globally!) As suggested in https://www.postfix.org/MILTER_README.html#workarounds
This way, 8-bit messages will not be converted to 7-bit QP or base64 when going through Mailman, and arrive intact at 8BITMIME capable recipients.
(After Mailman, Postfix' smtp client will still convert messages to 7-bit when delivering to non-8BITMIME capable recipients, which will still break DKIM validation for them, but non-8BITMIME capable DKIM-validators will have issues with a *lot* of mail anyway, forwarded or not.)
Geert
On Mon, Oct 31, 2022 at 00:04:29 +0100, Jaron Kent-Dobias wrote:
On Sunday, 30 October 2022 at 23:57 (+0100), Jaron Kent-Dobias wrote:
Confirmation: when Arch Linux forwards a base8 encoded email to the list, it mangles the DKIM. It does appear to be an Arch problem!
One last email: what the lists are specifically doing is rewriting 8bit encoded emails in a base64 encoding.
From the email in my sent folder:
Content-Transfer-Encoding: 8bit
From the email I received from the list:
Content-Transfer-Encoding: base64
Rewriting the body in a new encoding breaks DKIM.
Jaron
On 31.10.2022 15.11, Geert Hendrickx wrote:
In the chain sender => list.archlinux.org => list members, I suspect it's Mailman (on the Arch list server) that does not support 8BITMIME; since OpenDKIM on that server succesfully verified the DKIM signature, the mail arrived there as 8bit, but is being distributed to list members as 7-bit.
Could the Arch listserver set "disable_mime_output_conversion=yes" in its master.cf at the point it is handing over mail to Mailman? (not globally!) As suggested in https://www.postfix.org/MILTER_README.html#workarounds
This way, 8-bit messages will not be converted to 7-bit QP or base64 when going through Mailman, and arrive intact at 8BITMIME capable recipients.
(After Mailman, Postfix' smtp client will still convert messages to 7-bit when delivering to non-8BITMIME capable recipients, which will still break DKIM validation for them, but non-8BITMIME capable DKIM-validators will have issues with a *lot* of mail anyway, forwarded or not.)
Geert
On Mon, Oct 31, 2022 at 00:04:29 +0100, Jaron Kent-Dobias wrote:
On Sunday, 30 October 2022 at 23:57 (+0100), Jaron Kent-Dobias wrote:
Confirmation: when Arch Linux forwards a base8 encoded email to the list, it mangles the DKIM. It does appear to be an Arch problem!
One last email: what the lists are specifically doing is rewriting 8bit encoded emails in a base64 encoding.
From the email in my sent folder:
Content-Transfer-Encoding: 8bit
From the email I received from the list:
Content-Transfer-Encoding: base64
Rewriting the body in a new encoding breaks DKIM.
Jaron
Hi,
Thanks for investigating and reporting the issue! Me and foutrelis[1]has been doing some debugging and after upgrading mailman3 from 3.3.5-6 -> 3.3.7-1, we are unable to reproduce the issue.
Looking at the changelog[2] for mailman3 3.3.7, we suspect the issue was fixed as part of [3] and [4].
We are aware of one open issue[5] which can break the DKIM signature and with some luck it will be fixed in the future.
[1] https://archlinux.org/people/developers/#foutrelis [2] https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/docs/NEWS.rst [3] https://gitlab.com/mailman/mailman/-/issues/965 [4] https://gitlab.com/mailman/mailman/-/issues/967 [5] https://gitlab.com/mailman/mailman/-/issues/636
P.S. Adding a emoji 😎 to verify that this is indeed fixed at your end.
Cheers, Kristian Klausen Arch Linux DevOps
On Sun, Nov 13, 2022 at 13:43:03 +0100, Kristian Klausen wrote:
P.S. Adding a emoji 😎 to verify that this is indeed fixed at your end.
Hi
Your message was base64-encoded (I think by GnuPG), so passes OK anyway.
Trying again with this shorter message. Tħåñk ÿóú спасибо.
Geert
On Sun, Nov 13, 2022 at 15:07:20 +0100, Geert Hendrickx wrote:
On Sun, Nov 13, 2022 at 13:43:03 +0100, Kristian Klausen wrote:
P.S. Adding a emoji 😎 to verify that this is indeed fixed at your end.
Hi
Your message was base64-encoded (I think by GnuPG), so passes OK anyway.
Trying again with this shorter message. Tħåñk ÿóú спасибо.
This one came through as Content-Transfer-Encoding: 8bit and with dkim=pass.
Thanks!
Geert
On 11/13/22 7:43 AM, Kristian Klausen wrote:
On 31.10.2022 15.11, Geert Hendrickx wrote: Hi,
Thanks for investigating and reporting the issue! Me and foutrelis[1]has been doing some debugging and after upgrading mailman3 from 3.3.5-6 -> 3.3.7-1, we are unable to reproduce the issue.
Looking at the changelog[2] for mailman3 3.3.7, we suspect the issue was fixed as part of [3] and [4].
Thanks very much for looking into this and upgrading the mailman package, Kristian. Much appreciated!
DR
arch-general@lists.archlinux.org