[arch-dev-public] mailman hiccup

[RedShift]

Geoffroy Carrier wrote:
> On Tue, Jan 20, 2009 at 09:10, RedShift <redshift at pandora.be> wrote:
>> I've been working on my own mailing list manager, it's completely written in
>> PHP and uses SQL as a backend, but at this point a bit specific to our
>> setup. Because our company is all about open source I intend on releasing it
>> as such. It's in need of an overhaul anyway, I'll get hacking at it again
>> ASAP.
> 
> I'd love to be an early adopter :)
> 
> Plus I'd be happy to contribute...
> 
> 


Ok so here's what I've got so far:


When I was designing my anti-SPAM filter, I noticed you're just running a series of code after each other. To keep everything nice and tidy I split up the various routines into small pieces of code designed to do one thing and then pass on to the next filter. My project has the codename "redsky". Yes, it's cliché but I couldn't think of anything else.

By itself, redsky is pretty useless. It's only capable of running a series of filters after each other. The real power is in the filters itself. The 3 filters that come shipped with redsky are designed to interface with another piece of software called proxsmtpd (but can be used in other ways as well, they just use STDIN and STDOUT), but there's no reasons you could write your own input/output filters and make redsky a mini-SMTP server.


Here are the packages that are of interest:

* phpemailtoolbox (http://aur.archlinux.org/packages.php?ID=22237)

This is a small set of email related functions and classes. It is required by redsky.

* redsky (http://aur.archlinux.org/packages.php?ID=22261)

This is the main redsky code. It contains three filters:
* redsky_read: this reads the email in question from STDIN. It is designed for interfacing with proxsmtpd, but can be used in a generic manner as well.
* redsky_discard: when the discard flag is set on an email, this filter will log this event.
* redsky_out: this outputs the email to STDOUT, again designed to interface with proxsmtpd but is suitable for general purpose as well. In the mailinglist

* redsky-filters (http://aur.archlinux.org/packages.php?ID=24393)

This is a set of complementary filters for redsky, it has for example SMTP blacklist checking, whitelisting, autoreplies, etc... They are mainly designed to use in a postfix content scanner context. Only the redsky_sqlconn filter is required for mailinglist manager.

* mlm-filters (http://aur.archlinux.org/packages.php?ID=24399)

This is the most interesting set of filters. Mlm stands for Mailing List Manager. There's a queueing part and a sending part.

For the queueing part:
* mlm_getinfo: Fetch information for a certain list using SQL
* mlm_loopdetect: Check if a message is looping or not
* mlm_bouncehandler: If the message received was sent to a bounce address, store it in an SQL table
* mlm_getsubscribers: Get the subscribers from SQL for the current list
* mlm_access: Determine if a sender has access to the list. It currently supports 4 methods: use an ACL, only allow subscribers, require a passphrase or no limitations
* mlm_subjecttag: Tag the subject if the mailinglist is configured for that
* mlm_addfooter: Add a footer to the emails if the mailinglist is configured for that
* mlm_queuer: Write a spoolfile and put all the recipients into an SQL table

For the sending part:
* mlm_singleinstance: Only allow a single instance of redsky to be run
* mlm_queuebuilder: Build a queue of messages to be sent and their respective recipients
* mlm_queuerunner: Process the earlier built queue using sendmail to send messages


redsky is written to support multiple configurations. In the MLM case, you would have two configuration files (examples are shipped with mlm-filters), one for the queuer and one for the sender. The queuer would be called by the MTA while to sender could be executing with a cronjob. Use -c to supply an alternate config, for example:

redsky -c /etc/redsky.queuerunner.config.php



You will notice that the redsky-filters and mlm-filters come shipped with example configuration files. You will find them in /usr/share/php/redsky/. Some filters depend on each other, so it is important in which order they are executed. For example, the mlm_access cannot determine if the sender is a subscriber when mlm_getsubscribers hasn't run first, because it uses the information from mlm_getsubscribers for this.


Now maybe the hardest part, integration with an email server. I'll only handle postfix. Our entire infrastructure is based on MySQL, however all redsky code is written to use PDO, so it should be compatible with any databaseserver that PDO supports.

You can get our database schema from http://projects.webmind.be/redsky/mail.sql

In postfix terms, you would add entries to your alias table to send certain addresses to the "mlm". The examples below are made to work with the database schema above, but all queries can be modified in the configuration files.

virtual_alias_maps = mysql:/etc/postfix/mysql/lists.cf

lists.cf:

user = USER
password = PASSWORD
hosts = HOST
dbname = MAILDB

query = SELECT
			'mlm' AS goto
		FROM
			lists
        LEFT JOIN
        	domains
        ON
        	domains.domain_id = lists.domain_id
        WHERE
		(lists.list_post_address = '%u'	OR
		lists.list_bounce_address = '%u')	AND
		domains.domain_name = '%d'		AND
		domains.domain_type = 'virtual'		AND
		domains.domain_enable_email	= 'yes'

The above will send all mail sent to a mailing list address to 'mlm'. 

Now postfix knows it needs to send any list emails to mlm. Now we need to tell postfix that all email sent to the local 'mlm' address to be piped to our mlm program. For this we create a transport table, that has 'mlm at localhost.localdomain' transported to the 'mlm' destination.

Now we have to define what the mlm destination is. Open up master.cf and add:

mlm	unix	-	n	n	-	-	pipe
	user=nobody	flags=O argv=/usr/bin/redsky -c /etc/redsky.queuer.config.php

Note the O flag, this will add an X-Original-To header, which is required to identify wich original address the message was sent to.


To allow the messages to be distributed, just call /usr/bin/redsky -c /etc/redsky.runner.config.php

Ok, all this information was probably hard to swallow. I will try to answer any questions you have regarding the implementation. If you want to ask me a direct question you can find me on IRC, freenode with the nick RedShift. I am in #archlinux.

It's also easier if you start by testing it out directly from the shell instead of immediatly integrating it into your MTA.


Best regards,

Glenn