Linux Server Setup: Part 7 - Mail Servers

Updated: September 14, 2010

An MTA functions like a post office, processing incoming and outgoing mail. For outgoing mail, it relies on the Simple Mail Transfer Protocol (SMTP). In order to send mail, the user must be authorized to contact the SMTP service. This may or may not include authentication. Once the user is approved, the message is transported to a queue where it awaits delivery. The MTA then contacts the domain specified in the email address and attempts to deliver the message. All of this can take place very quickly and appear almost instantaneous to the user.

Similarly, when mail is received, it is filed to the specified user's account. If the user is not found, the mail server will determine how the message should be routed according to any instructions specified in configuration (i.e. a catch-all address, a return to sender, or simple deletion). Once it is stored in the user's inbox, the user will need to use an email client to contact the POP or IMAP server, which is often a separate program from the MTA.

Many mail servers support two different types of mailboxes. One is called mbox and stores all of the messages for an account in a single text file. In contrast, a maildir mailbox will store messages in individual files inside of a mailbox directory.

The most common port for SMTP (outgoing mail) is 25, although the system administrator can change this setting in the mail server configuration file (often found within the /etc/mail directory in Linux). Any changes to system configuration require the mail server to be restarted before they take effect. If the port is changed, users must configure their email clients to reflect those changes.


One reason many individuals and organizations have switched to cloud-based email services is that running a mail server requires a great amount of security configuration and maintenance. There are three primary areas of security concern: SMTP access, spam control, and virus protection. All three require consistent monitoring and management in order to ensure the security of email accounts, messages, and the server as a whole.

SMTP access is generally limited to users on the same domain as the mail server. SMTP servers that allow unrestricted access are called open relays and are prime targets for spammers. Furthermore, most DNSBL services will blacklist a server with an open relay, which creates delivery problems for its users.

One way to prevent an open relay or even the exploitation of a closed one is to require user authentication before mail transport. If a user is required to login with a username and password before the account can send a message, this greatly reduces the likelihood that a spammer can take advantage of the mail server. The only other methods a spammer could use for intrusion would be to find a software vulnerability or guess the user's password. One way to prevent security breeches during authentication is to use TLS or SSL encryption.

While closing an open relay will reduce the likelihood of spam originating from the server, only anti-spam software can prevent spam from being delivered to user accounts. There are two primary methods for controlling spam, depending on the software. The first is to filter messages received by the MTA and quarantine any that fail to pass the spam check. The second method involves using a proxy that sits between the Internet and the MTA, filtering out likely spam messages before the MTA has a chance to receive them.

Fighting spam usually involves a combination of filtering techniques, including DNSBL (blacklisting), whitelists, filter rules that study word usage, bayesian filtering, checksum detection, and online databases. No one technique is perfect, and some, such as bayesian filtering, require the user or system administrator to train the software to stop spam and allow ham (legitimate emails).

Viruses are most often distributed through email. Therefore, it is critical that a mail server have some type of virus scanning in place. Many scanners do more than just virus detection and may even rely on third-party software to actually detect the viruses. MailScanner, for example, relies on SpamAssassin for spam detection and any of a number of anti-virus software tools.

Although Linux is commonly believed to be virus-free and not in danger of being infected, a Linux mail server must still have anti-virus software in place to prevent Windows viruses from being transported to users.

Related Categories
Featured Research
  • Securing Enterprise Information Technology


  • Office365 Adoption eGuide


  • Okta Directory Integration


  • Top 8 Identity and Access Management Challenges with Your SaaS Applications


  • Better BYOD with Pulse Secure and MDM Partners
