OS X Server: Microsoft Outlook clients might not be able to send mail due to spam filter settings

Microsoft Outlook users might not be able to connect to an email server on OS X Server if your spam filter is configured in a certain way.

Users who use Microsoft Outlook as their email client and who connect to an email server hosted from Mac OS X Server v10.6 or later might not be able to send mail if the spam filter is configured in a certain way. The following (or similar) alert is returned to Outlook clients:

The message could not be sent because one of the recipients was rejected by the server. The rejected email address was "recipient@example.com" is subject "example", account: "mail.example.com" , server "mail.example.com", protocol: SMTP, server response: "504 5.5.2 < hostname > : Helo command rejected: need fully qualified host name", port: 25, secure (SSL): no, server error: 504, error number: 0x800CCC79.

This issue happens when the OS X Server Mail service (postfix) is configured to require a fully qualified hostname from SMTP clients. This setting is enabled when you turn on spam filtering in Mac OS X Server v10.6 or later.

The fully qualified hostname requirement can be configured separately from other spam filter settings by editing a configuration file. Any modification of a security-related setting should be evaluated prior to making the change.

Once you have evaluated the change, you can use the following steps to implement it. Use these guidelines when editing a configuration file.

For OS X Server (Mountain Lion and later)

1. In /Library/Server/Mail/Config/postfix/main.cf, locate the smtpd_helo_restrictions setting.

2. Remove "reject_non_fqdn_helo_hostname" from the list of settings.

3. Restart the Mail service.

For OS X Lion Server and Mac OS X Server v10.6.x

1. In /etc/postfix/main.cf, locate the smtpd_helo_restrictions setting.

2. Remove "reject_non_fqdn_helo_hostname" from the list of settings.

3. Restart the Mail service.