On Wed, Nov 13, 2019 at 10:15:37PM +0100, Jelle van der Waa <jelle@vdwaa.nl> wrote:

I've noticed that the bugtracker is super slow when assigning bugs. It takes around ~6-8 seconds to save a task I'm wondering if this is due to one of or recent changes either the mail relay or fail2ban. Anyone have an idea how to figure out if it's due to the mail relay?

As explained on IRC, mails are no longer sent via port 10027, but via 25 now. 25 is configured to scan mails with spamassassin before accepting them and that adds some delay which piles up depending on how many users flyspray sends notifications to. Right now 10027 is disabled on the non-mailserver machines. I believe that is because the comment I put in master.cf says that it is a port for DKIM signature injection and Philip wanted to centralize DKIM signing on orion for easier handling. In that process he likely cleaned up the now supposedly unneeded port. Services were then switched to port 25 instead, but 25 is usually used for mail coming from the internet. Since you can't trust such mail, we scan it for spam. Because you still cannot trust incoming mails, you want to scan it before you accept it because otherwise you can't be sure who you reach if you send a notification to the "sender" (Return-Path) of the mail. The sender could easily be faked. Scanning during delivery obviously leads to delays as we experience now. This is usually not a problem for regular emails that are sent from mail servers because those can send multiple mails at the same time. Flyspray however likely only sends them serially and not in parallel, thus the delay becomes noticeable. It is easily possible to accept mails first and scan them later, but you can only do that if you trust that the sender address of the mail is legit. Otherwise you open yourself up to "backscatter". I thus advise against changing the current handling of port 25 unless that change also restricts the port to localhost/trusted senders. I'd suggest that we fix the delay by putting 10027 back in action on all hosts even if they do not do the DKIM signing themselves. We can then use that port to let services (flyspray) deliver their mail as they did before. This will stop mails from being scanned for spam on the originating host, but that's fine because spam scanning also currently happens on orion too even if e.g. apollo already scanned the mail once. In this case we don't have to fear backscatter because the mails generated by our services use trusted Return-Path values so the bounce mails for spam mails will not end up in some random person's inbox. I think all you need to do to make this happen is to remove the {% if %} condition around the service definition for port 10027 in roles/postfix/templates/master.cf.j2. You might also want to update the comment in the process so that it's clear what the port is used by. Florian