[arch-devops] Flyspray (bugs.archlinux.org) slow when saving/creating

[Florian Pritz]

On Wed, Nov 13, 2019 at 10:15:37PM +0100, Jelle van der Waa <jelle at vdwaa.nl> wrote:
> I've noticed that the bugtracker is super slow when assigning bugs. It
> takes around ~6-8 seconds to save a task I'm wondering if this is due to
> one of or recent changes either the mail relay or fail2ban. Anyone have
> an idea how to figure out if it's due to the mail relay?

As explained on IRC, mails are no longer sent via port 10027, but via 25
now. 25 is configured to scan mails with spamassassin before accepting
them and that adds some delay which piles up depending on how many
users flyspray sends notifications to. 

Right now 10027 is disabled on the non-mailserver machines. I believe
that is because the comment I put in master.cf says that it is a port
for DKIM signature injection and Philip wanted to centralize DKIM
signing on orion for easier handling. In that process he likely cleaned
up the now supposedly unneeded port.

Services were then switched to port 25 instead, but 25 is usually used
for mail coming from the internet. Since you can't trust such mail, we
scan it for spam. Because you still cannot trust incoming mails, you
want to scan it before you accept it because otherwise you can't be sure
who you reach if you send a notification to the "sender" (Return-Path)
of the mail. The sender could easily be faked. Scanning during delivery
obviously leads to delays as we experience now. This is usually not a
problem for regular emails that are sent from mail servers because those
can send multiple mails at the same time. Flyspray however likely only
sends them serially and not in parallel, thus the delay becomes
noticeable.

It is easily possible to accept mails first and scan them later, but you
can only do that if you trust that the sender address of the mail is
legit. Otherwise you open yourself up to "backscatter". I thus advise
against changing the current handling of port 25 unless that change also
restricts the port to localhost/trusted senders.

I'd suggest that we fix the delay by putting 10027 back in action on all
hosts even if they do not do the DKIM signing themselves. We can then
use that port to let services (flyspray) deliver their mail as they did
before. This will stop mails from being scanned for spam on the
originating host, but that's fine because spam scanning also currently
happens on orion too even if e.g. apollo already scanned the mail once.
In this case we don't have to fear backscatter because the mails
generated by our services use trusted Return-Path values so the bounce
mails for spam mails will not end up in some random person's inbox.

I think all you need to do to make this happen is to remove the {% if %}
condition around the service definition for port 10027 in
roles/postfix/templates/master.cf.j2. You might also want to update the
comment in the process so that it's clear what the port is used by.

Florian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20191115/30a965ba/attachment.sig>