howto sync mailman from plone

export members

The script below is basically a rewrite from a script on plone.org with added functionality:

• choose which columns to export (needed for mailman: it wants only the "email" field)
• choose with or without column names as first record (mailman requires without header record)
• choose formatting of output
• filter on column values (enables unsubscription, see below)

The last one, filtering on column values, is crucial in enabling the option to unsubscribe. Which is, of course, what distinguishes an opt-in newsletter from spam.

The example implementation here requires that one of four custom checkboxes is activated. If users indicate no interest in any of our four topic areas, they're unsubscribed from the newsletter. So, here's the script (also available as download).

## Script (Python) "export_members"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=fieldlist='',fieldsep=',',fieldreq='',op='and',noheader=False
##title=generate a csv export of member data
##
REQUEST=context.REQUEST
from zLOG import LOG, INFO
from Products.CMFCore.utils import getToolByName

## see http://plone.org/documentation/how-to/export-member-data-to-csv

## fieldlist:   which member properties to export
## fieldsep:    which character used to split/join lists (default: comma)
## fieldreq:    which field values must evaluate True (filter on field values)
## reqop:       evaluate required fields with AND or OR

## export_members?fieldsep=; will give fieldsep==''
if not fieldsep: fieldsep = ';'

if fieldlist != '':
    memberProperties = fieldlist.split(fieldsep)
else:
    memberProperties = ('id','email','fullname',
                        ## insert your additional memberdata here                        
                        'geslacht','bedrijfsnaam','functie','telefoon', # example custom fields
                        'nieuws_plus','nieuws_cm','nieuws_pd','nieuws_tt' # custom topic checkboxes
                        )
lines = []

## double negative so any non-default value triggers
if not noheader:
    lines.append(fieldsep.join(memberProperties))
    
membership = getToolByName(context,'portal_membership')

for memberId in membership.listMemberIds():
    member = membership.getMemberById(memberId)
    if fieldreq != '':
        requiredValues = [member.getProperty(x) for x in fieldreq.split(fieldsep)]
        if op == 'or':
            memberok = reduce(lambda x,y: x or y, requiredValues, False ) # any True succeeds
        else:
            memberok = reduce(lambda x,y: x and y, requiredValues, True ) # any False fails
        ## uncomment to debug
        #lines.append(str(op)+':'+','.join((str(x) for x in requiredValues)) + '=%s' % memberok)            
        if not memberok: continue
    ## replace all commas in values, create comma-separated-value string, append to output
    lines.append(fieldsep.join( (str(member.getProperty(x)).replace(fieldsep,' ') for x in memberProperties) ))

return 'n'.join(lines) + 'n'

import into mailman

To import the Plone member export into Mailman, you'll have to feed the export into mailman's sync_members utility. The export script above by default produces a full all-columns listing of the Members database that's unsuitable for Mailman import. We need just the email column, with no header record, and filtered on special subscription properties. Also, the script needs manager access to the Plone site to protect our Member data.

Putting all of this directly into a cron record is possible; however this will log your password in plaintext to the syslog. Also, you'll have to escape all ampersands for the query arguments. A nicer solution uses Apache rewrite:

    ## mailman export
    RewriteRule ^/sync_members /export_members?fieldlist=email&op=or&fieldreq=nieuws_plus,nieuws_cm,nieuws_pd,nieuws_tt&noheader=1

Now we have two export URLs:

export_members

the full export script as documented above.

sync_members

a convenience URL that calls export_members in it's mailman configuration:

• just one field: email
• require that at least one of four topic areas is checked
• suppress the column header record

The actual syncing is performed by invoking from cron, the script below:

/usr/bin/snarf admin:PASSWORD@PLONESITE/sync_members - 
| /usr/sbin/sync_members -w=no -a=no -f - MAILINGLIST

This fetches the member listing, spits it out on STDOUT and feeds it to the mailman importer with appropriate options. You'll have to substitute the correct admin user, password, site URL and mailing list name of course. YMMV.

custom checkbox gotcha's

As mentioned above, this setup uses some custom topic subscription checkboxes in the personalisation page for members to subscribe/unsubscribe to the mailing list. Checkboxes have a nasty drawback: an unchecked (or disabled) checkbox is not sent as a name=value pair in the form submission. You don't get somevar=False. You get nothing. Consequently, your custom property remains unchanged since there's no input to change it, unless you create a specific workaround.

First, create the topic checkboxes through GenericSetup. In profiles/default/memberdata_properties.xml define your custom member fields plus topic areas.

Implement the additional fields in additional_memberdata.pt. This template, not documented here, wil automatically be called from personalize_form.cpt if it exists. You can also call it from join_form.pt if you want to.

Now, to handle the checkboxes correctly, customize personalize.cpy, by interpreting the absence of one of the variables as meaning variable=False. Then store the newly parsed values. Insert the following code block into personalize.cpy:

    
if nieuws_plus is None and REQUEST is not None:
    nieuws_plus=0
else:
    nieuws_plus=1
REQUEST.set('nieuws_cm', nieuws_cm)
if nieuws_cm is None and REQUEST is not None:
    nieuws_cm=0
else:
    nieuws_cm=1
REQUEST.set('nieuws_pd', nieuws_pd)
if nieuws_pd is None and REQUEST is not None:
    nieuws_pd=0
else:
    nieuws_pd=1
REQUEST.set('nieuws_tt', nieuws_tt)
if nieuws_tt is None and REQUEST is not None:
    nieuws_tt=0
else:
    nieuws_tt=1
REQUEST.set('nieuws_tt', nieuws_tt)

member.setProperties(nieuws_plus = nieuws_plus,
                     nieuws_cm = nieuws_cm,
                     nieuws_pd = nieuws_pd,
                     nieuws_tt = nieuws_tt)

done!

To recap: you now have member profiles with four topic checkboxes. If any of these checkboxes is checked, the member's email address will be subscribed (or remain subscribed) to the Mailman list. If all four checkboxes are unchecked, the member's email address is omitted from the export and will be unsubscribed by the Mailman sync_members import utility.

TODO: setting Mailman topics from the topic checkboxes. I haven't figured out how to integrate that... Any suggestions?