What Is a Mail Server? Setup, SMTP/IMAP and Domain Mail Guide

A mail server is the mostly invisible but mission-critical software stack that makes email work at internet scale. Every time a user clicks “send,” dozens of processes, certificates, DNS lookups, and protocol exchanges land on the mail server’s shoulders. This guide doesn’t stop at conceptual definitions; it walks you through how to build your own mail server from scratch, which DNS records you must publish, how to protect deliverability, and the real-world difference between buying managed mail hosting and running the server under your own control — all backed by 2026 data.

Related guides: What is DNS and how to change settings · Domain names and WHOIS lookup · Free SSL with Let’s Encrypt · Nginx configuration guide · VPS security hardening · Linux server administration basics

What Is a Mail Server? A Definition From the Roots Up

A mail server is the collection of programs that accept, relay, store, and deliver email messages over the internet. There is no single piece of software at its core; a mail server is actually an architecture of at least three distinct components: the MTA (Mail Transfer Agent — moves mail from server to server), the MDA (Mail Delivery Agent — writes incoming messages into the right user’s mailbox), and the MUA (Mail User Agent — the user-facing client like Outlook, Thunderbird, or webmail).

The backbone underneath this architecture is a handful of RFCs published since 1982: RFC 5321 (SMTP), RFC 5322 (Internet Message Format), RFC 3501 (IMAP4rev1). SMTP’s original design goal was store-and-forward; it still operates on that same fundamental logic today — but a dozen layers of TLS, authentication, antispam, and DKIM have been stacked on top.

How a Mail Server Works: The Journey of an Email

Most users have no idea what unfolds behind the scenes when they hit send. In reality, every message passes through at least six checkpoints in roughly 200 milliseconds to 30 seconds. From left to right: sender MUA, sender submission server (port 587), sender outbound MTA, recipient domain’s MX server (port 25), recipient IMAP/POP3 store, and finally the recipient MUA. If any link in that chain breaks, the message either waits, bounces back, or lands in the spam folder.

• 1. Submission — After authenticating the user, the MUA hands the message to the submission server on port 587.
• 2. Outbound queue — The MTA queues the message and runs an MX DNS lookup for the recipient domain.
• 3. SMTP conversation — The sending MTA connects to the recipient MX on port 25 and delivers the message via EHLO, MAIL FROM, RCPT TO, and DATA.
• 4. Antispam & auth check — The receiving server runs SPF, DKIM, DMARC, RBL, and content-scoring tests.
• 5. Local delivery — The MDA writes the message into the user’s mailbox (Maildir or mbox format).
• 6. Retrieval — The user pulls the message via IMAP or POP3, or reads it through webmail directly on the server.

SMTP, IMAP, and POP3: Which Protocol Does What?

Mixing up the three email protocols is a common mistake. The short version: SMTP is for sending; IMAP and POP3 are for receiving. SMTP works on a push model, IMAP/POP3 on a pull model.

• SMTP (Simple Mail Transfer Protocol, RFC 5321) — Carries messages from one server to another. Submission uses port 587 (STARTTLS), server-to-server transfer uses port 25, and legacy implicit TLS uses 465.
• IMAP (Internet Message Access Protocol, RFC 3501) — Keeps messages on the server, syncs folder structure and read/unread flags. The default choice for accessing mail from multiple devices. Port 143 (STARTTLS) or 993 (TLS).
• POP3 (Post Office Protocol v3, RFC 1939) — Downloads messages from the server, usually deleting them in the process. Still useful for offline-archive scenarios. Port 110 (STARTTLS) or 995 (TLS).
• Always use submission — ISPs commonly block port 25. For Outlook/Thunderbird, configure 587 + STARTTLS without exception.
• Implicit TLS vs STARTTLS — Implicit TLS encrypts the connection from the moment it opens; STARTTLS starts in plain text and upgrades to encryption. The practical difference is small in modern clients, but auditors prefer implicit TLS.

Domain Mail Server: A Mail System That Runs on Your Own Domain

A “domain mail server” — commonly called “branded email” — lets you operate addresses in the form mailbox@yourcompany.com. The difference between info@gmail.com and info@yourcompany.com isn’t just aesthetic; the latter signals trust, professionalism, and brand consistency, and measurably increases customer response rates.

Three building blocks are essential to set up a domain mail server: a domain name (see our domain and WHOIS guide), the correct DNS records (MX, SPF, DKIM, DMARC), and either a self-hosted mail server or a managed mail hosting product. If there’s a gap between domain registration and mail setup, expect 4–48 hours for full DNS propagation.

The DNS Side: MX, A, SPF, DKIM, and DMARC Records

No matter how well-tuned your mail server is, if the DNS records are wrong your messages will either never arrive or end up in spam. You must know these five record types by heart: MX (the recipient server’s address), A/AAAA (the IP of your mail host), SPF (TXT — who is allowed to send on your behalf), DKIM (TXT — cryptographically signs messages), and DMARC (TXT — defines what to do when SPF/DKIM fails).

; Zone file example (BIND syntax) - yourcompany.com
; -----------------------------------------------
@ IN MX 10 mail.yourcompany.com.
mail IN A 195.175.10.20
autodiscover IN CNAME mail.yourcompany.com.
autoconfig IN CNAME mail.yourcompany.com.

; SPF - only allow your own mail server and relay
@ IN TXT "v=spf1 mx a:mail.yourcompany.com ip4:195.175.10.20 -all"

; DKIM - selector name 'mail' (generated by Postfix/OpenDKIM)
mail._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD...QAB"

; DMARC - start with quarantine and collect reports
_dmarc IN TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@yourcompany.com; ruf=mailto:dmarc@yourcompany.com; pct=100; adkim=s; aspf=s"

In SPF, the trailing -all means “hard fail” — messages from IPs not on the list are rejected. Start with ~all (soft fail) in a test environment, then tighten to -all once you’re confident your records are right. The DKIM selector name (mail in the example) must match the key pair generated on the server. For DMARC, it’s a good idea to begin with p=none (report only, take no action); after 1–2 weeks of data, move to p=quarantine, then to p=reject.

Reverse DNS (PTR), HELO, and FCrDNS

Most spam filters dock points if your mail server has no PTR (reverse DNS) record, or if its HELO banner doesn’t match the PTR. PTR records are typically created in your IP provider’s panel; most VPS providers offer a “Reverse DNS” or “rDNS” field in the user interface. Forward-confirmed reverse DNS (FCrDNS) means the A record for mail.yourcompany.com resolves to x.y.z.w and that same IP’s PTR points back to mail.yourcompany.com. If both directions don’t agree, Gmail, Outlook, and Yahoo will silently filter your mail.

# PTR and FCrDNS check
dig +short mail.yourcompany.com A
# 195.175.10.20

dig +short -x 195.175.10.20
# mail.yourcompany.com.

# Match the Postfix HELO banner with the PTR
postconf -e 'myhostname = mail.yourcompany.com'
postconf -e 'smtp_helo_name = mail.yourcompany.com'

# A thorough tool to test your mail server
# Send a test message to -> https://www.mail-tester.com
# Returns DKIM/SPF/DMARC/PTR/blacklist score

Setting Up a Mail Server: Two Paths

Behind the search query “how to set up a mail server” sit two completely different intents: some people want “buy ready-made, just use it”; others want “I’m going to roll up my sleeves and run my own server.” Both are legitimate choices, but the costs and operational burden differ.

• Managed mail hosting / corporate email plan — You rent per-user domain mailboxes from a multi-tenant server operated by a provider. DNS, antispam, backups, updates, and blacklist management are the provider’s responsibility.
• Self-hosted mail server — You install Postfix + Dovecot, iRedMail, Mailcow, or Mail-in-a-Box on your own VPS and run all of the infrastructure yourself. You get full control, but the operational load is real.

For small teams, self-hosting is a romantic but risky choice: blacklist avoidance, IP reputation, and antispam tuning add up to roughly 4–6 hours of weekly maintenance on their own. Building IP reputation today can take years; one compromised account can halt your outbound mail for weeks. Our recommendation: teams smaller than 50 should use managed mail hosting and configure SPF/DKIM/DMARC correctly on top; larger teams or those with regulatory obligations should move to self-hosting.

The Self-Hosted Stack: What Will You Install?

The open-source mail-server ecosystem is mature. As of 2026, the most common combinations are:

• Postfix + Dovecot + Rspamd + OpenDKIM — The classic “build from components” approach. Maximum flexibility, minimum magic. Best support for debugging and performance tuning.
• iRedMail — Installs Postfix + Dovecot + Amavis + SpamAssassin with a single script. Supports CentOS/Ubuntu/Debian, with a free Community edition. Very practical for small-to-medium scale.
• Mailcow — A dockerized mail server. Brings up Postfix, Dovecot, Rspamd, ClamAV, SOGo (groupware), Nginx, Redis, and MariaDB with a single docker-compose. The standout choice for modern teams. Read our Docker Compose guide for context.
• Mail-in-a-Box — Promises “a mail server with one command.” Better suited to individual or small-scale use. Customization is hard.
• Modoboa — A Python-based admin panel integrated with Postfix/Dovecot/Rspamd. Attractive for teams that want a simple web UI.
• Zimbra Open Source / Carbonio — Groupware-focused, bundling calendar, contacts, and file sharing. An open-source alternative to Exchange.

Step 1: Server Prep and Hardening

For the rest of this guide we’ll walk through the most common and transparent approach — Postfix + Dovecot + Rspamd + OpenDKIM — on Ubuntu 24.04 LTS, step by step. Before installing the mail server, lay down the security basics on your VPS; for a deeper dive, read our VPS security hardening and Fail2ban guide. Mail servers attract some of the highest brute-force traffic on the internet; 50–100 failed SSH login attempts per minute is normal.

# Update the system and install base packages
sudo apt update && sudo apt -y full-upgrade
sudo apt -y install ufw fail2ban unattended-upgrades \
 curl vim net-tools dnsutils

# Pin the hostname and FQDN
sudo hostnamectl set-hostname mail.yourcompany.com
echo '127.0.1.1 mail.yourcompany.com mail' | sudo tee -a /etc/hosts

# Enable automatic security updates
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Firewall - only the ports you need
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp # SSH (harden afterward)
sudo ufw allow 25/tcp # SMTP (server-to-server)
sudo ufw allow 587/tcp # Submission
sudo ufw allow 465/tcp # SMTPS (implicit TLS)
sudo ufw allow 143/tcp # IMAP STARTTLS
sudo ufw allow 993/tcp # IMAPS
sudo ufw allow 110/tcp # POP3 (optional)
sudo ufw allow 995/tcp # POP3S (optional)
sudo ufw allow 80/tcp # HTTP (Let's Encrypt)
sudo ufw allow 443/tcp # HTTPS (webmail)
sudo ufw enable

Step 2: Installing the TLS Certificate

A modern mail server must run with TLS. Let’s Encrypt provides free, auto-renewing certificates and is the most common choice for mail servers. For step-by-step configuration, see our Let’s Encrypt guide.

# Install certbot
sudo apt -y install certbot

# Standalone challenge instead of DNS-01 (keep port 80 free)
sudo certbot certonly --standalone \
 -d mail.yourcompany.com \
 --email admin@yourcompany.com \
 --agree-tos --non-interactive

# Reload Postfix/Dovecot after every renewal
sudo tee /etc/letsencrypt/renewal-hooks/deploy/mail-reload.sh <<'EOF'
#!/bin/bash
systemctl reload postfix
systemctl reload dovecot
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/mail-reload.sh

Step 3: Installing and Configuring Postfix

Postfix, written by Wietse Venema in 1998 as a Sendmail alternative, is the most widely deployed MTA on the internet today. Its modular design, auditable configuration, and strong security record make it a fit for everything from a small office server to ISP-scale infrastructure.

# Installation
sudo DEBIAN_FRONTEND=noninteractive apt -y install \
 postfix postfix-mysql postfix-pcre

# During install pick 'Internet Site' and enter mail.yourcompany.com

# /etc/postfix/main.cf - production-ready example
myhostname = mail.yourcompany.com
mydomain = yourcompany.com
myorigin = $mydomain
inet_interfaces = all
inet_protocols = ipv4
mydestination = localhost

# Virtual mailboxes - mailbox info lives in MySQL
virtual_mailbox_domains = mysql:/etc/postfix/mysql-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-mailboxes.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-aliases.cf
virtual_transport = lmtp:unix:private/dovecot-lmtp

# TLS - submission (587)
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourcompany.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourcompany.com/privkey.pem
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

# Submission auth (Dovecot SASL)
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

# Size limits
message_size_limit = 52428800 # accept 50 MB attachments
mailbox_size_limit = 0 # mailbox unlimited, quota is managed by Dovecot

# Antispam helpers
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks,
 reject_invalid_helo_hostname,
 reject_non_fqdn_helo_hostname
smtpd_sender_restrictions = reject_non_fqdn_sender,
 reject_unknown_sender_domain

In /etc/postfix/master.cf, enable the submission (587) and smtps (465) ports; both should always require SASL. Keep postfix.org/postconf.5 close at hand as a reference for every Postfix directive.

Step 4: Installing Dovecot (IMAP/POP3 + LMTP)

Dovecot takes on the MDA + IMAP/POP3 role where Postfix is the MTA. It receives messages from Postfix over LMTP, writes them to the user’s Maildir, and serves them over IMAP/POP3. Postfix also leverages Dovecot’s auth service.

sudo apt -y install dovecot-imapd dovecot-pop3d \
 dovecot-lmtpd dovecot-mysql dovecot-sieve

# Dovecot configuration files
# /etc/dovecot/dovecot.conf
# /etc/dovecot/conf.d/*.conf

# /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:/var/mail/vhosts/%d/%n
mail_privileged_group = mail

# /etc/dovecot/conf.d/10-auth.conf
disable_plaintext_auth = yes
auth_mechanisms = plain login
!include auth-sql.conf.ext

# /etc/dovecot/conf.d/10-master.conf
service imap-login {
 inet_listener imap { port = 143 }
 inet_listener imaps { port = 993; ssl = yes }
}
service lmtp {
 unix_listener /var/spool/postfix/private/dovecot-lmtp {
 mode = 0600
 user = postfix
 group = postfix
 }
}
service auth {
 unix_listener /var/spool/postfix/private/auth {
 mode = 0660
 user = postfix
 group = postfix
 }
}

# /etc/dovecot/conf.d/10-ssl.conf
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourcompany.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourcompany.com/privkey.pem
ssl_min_protocol = TLSv1.2

Step 5: DKIM Signing With OpenDKIM

DKIM applies a cryptographic signature to every outbound message; the receiving server fetches the public key from DNS and verifies the signature. It’s stronger than SPF because it also proves the message body wasn’t altered. OpenDKIM is the most widely used implementation that talks to Postfix as a milter.

sudo apt -y install opendkim opendkim-tools

# Create the key directory
sudo mkdir -p /etc/opendkim/keys/yourcompany.com
sudo opendkim-genkey -b 2048 \
 -d yourcompany.com \
 -D /etc/opendkim/keys/yourcompany.com \
 -s mail -v
sudo chown -R opendkim:opendkim /etc/opendkim
sudo chmod 600 /etc/opendkim/keys/yourcompany.com/mail.private

# The public key to publish in DNS:
sudo cat /etc/opendkim/keys/yourcompany.com/mail.txt
# Add the 'mail._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0..."' line
# as a TXT record in your DNS panel

# Bind to Postfix as a milter
sudo postconf -e 'milter_default_action = accept'
sudo postconf -e 'milter_protocol = 6'
sudo postconf -e 'smtpd_milters = inet:127.0.0.1:8891'
sudo postconf -e 'non_smtpd_milters = inet:127.0.0.1:8891'
sudo systemctl restart opendkim postfix

As of 2026, the minimum acceptable DKIM key size is 2048-bit RSA; 1024-bit should only appear on legacy systems. Best practice is to rotate the key once a year — name the selector mail2026, then mail2027 the following year.

Step 6: Antispam With Rspamd

Rspamd is the modern successor to the older SpamAssassin: written in C, dramatically faster, and bundling Bayes/neural-network learning, RBL checks, greylisting, rate-limiting, and hundreds of other rules. It integrates with Postfix as a milter.

# Install Rspamd
sudo apt -y install rspamd redis-server
sudo systemctl enable --now rspamd redis-server

# Postfix integration
sudo postconf -e 'smtpd_milters = inet:127.0.0.1:8891 inet:127.0.0.1:11332'
sudo postconf -e 'non_smtpd_milters = inet:127.0.0.1:8891 inet:127.0.0.1:11332'
sudo systemctl reload postfix

# Admin UI (default: localhost:11334)
# Set a password for remote access:
sudo tee /etc/rspamd/local.d/worker-controller.inc <<'EOF'
password = "$2$ka6...sample hash$";
bind_socket = "127.0.0.1:11334";
EOF
# Reach it via SSH tunnel: ssh -L 11334:localhost:11334 server

Redis is the data store Rspamd uses for Bayes data, fuzzy hashes, and rate limiting. For general Redis usage, see our Redis basics guide. Rspamd learns automatically from messages users move into the “spam” folder — you can wire up that training pipeline through IMAP Sieve filters.

Step 7: Webmail Setup (Roundcube)

Your users will need a webmail interface to read mail in the browser. The most common options are Roundcube (PHP), SnappyMail (PHP, lightweight), RainLoop (no longer maintained), and SOGo (groupware). Serving it through Nginx is the default step in most deployments — refer to our Nginx configuration guide.

# /etc/nginx/sites-available/webmail.yourcompany.com
server {
 listen 443 ssl http2;
 server_name webmail.yourcompany.com;
 root /var/www/roundcube;
 index index.php;

 ssl_certificate /etc/letsencrypt/live/webmail.yourcompany.com/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/webmail.yourcompany.com/privkey.pem;
 ssl_protocols TLSv1.2 TLSv1.3;

 add_header Strict-Transport-Security 'max-age=63072000; includeSubDomains' always;
 add_header X-Frame-Options DENY always;
 add_header X-Content-Type-Options nosniff always;

 location ~ \.php$ {
 fastcgi_pass unix:/run/php/php8.3-fpm.sock;
 fastcgi_index index.php;
 include fastcgi_params;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 }

 location ~ /\.(ht|git|env) { deny all; }
}

Testing and Validation

After the mail server is built, you absolutely must test it. Before going live with a real user, every item on this checklist needs to pass:

• Send a test message to mail-tester.com — aim for 10/10.
• Validate MX, SPF, DKIM, and DMARC records via mxtoolbox.com.
• Inspect your SMTP TLS handshake with checktls.com.
• Send test messages to Gmail, Outlook.com, Yahoo, and Yandex accounts and check the inbox/spam split.
• Test from the CLI with swaks --to test@gmail.com --from postmaster@yourcompany.com --server localhost.
• Open the message in Gmail, click “Show original,” and confirm SPF=PASS, DKIM=PASS, DMARC=PASS.
• Confirm DMARC reports start arriving at dmarc@yourcompany.com (1–3 days).

# Postfix queue status
mailq # see pending messages
postqueue -p
postqueue -f # force-flush the queue

# Tail the logs (live)
sudo journalctl -u postfix -f
sudo tail -f /var/log/mail.log

# Dovecot debug
sudo doveconf -n # print the active config
sudo dovecot -n # startup check

# OpenDKIM signature test
sudo opendkim-testkey -d yourcompany.com -s mail -vvv

# Test a specific DKIM signature offline
opendkim-testmsg < /tmp/saved-message.eml

Deliverability: The Invisible Battle

Even if your mail server is technically correct, your messages can still land in spam. The cause isn’t technical — it’s reputation. New IPs and new domains are treated with suspicion by Gmail, Microsoft 365, and Yahoo; building reputation takes weeks.

• IP warm-up — Send 50–100 messages/day in the first days from a new server; double the volume each week.
• List hygiene — Remove hard bounces from the list immediately. A bounce rate above 5% will dent reputation.
• One-click unsubscribe (List-Unsubscribe header) — Add RFC 8058 / RFC 2369 compliant List-Unsubscribe: <mailto:...>, <https:...> and List-Unsubscribe-Post: List-Unsubscribe=One-Click headers. Gmail/Yahoo have required this since 2024.
• Complaint rate (FBL feedback loop) — Keep it below 0.3%.
• Stream separation — Don’t only send marketing; route transactional and marketing traffic through different IPs/domains.
• BIMI — Combine DMARC p=quarantine+ with a BIMI record; your logo will appear next to the sender name in Gmail/Yahoo.
• Postmaster Tools — Register your domain at postmaster.google.com and postmaster.live.com to view reputation, spam rate, and auth data.

Blacklist (DNSBL) Monitoring

If your IP lands on Spamhaus ZEN, SORBS, Barracuda, UCEPROTECT, or Backscatterer, your messages start being rejected immediately. Compromised accounts, virus-infected endpoints, and poorly configured contact forms are common reasons for getting listed.

# Quick check for a single IP
for bl in zen.spamhaus.org bl.spamcop.net dnsbl.sorbs.net \
 b.barracudacentral.org dnsbl-1.uceprotect.net; do
 result=$(dig +short "20.10.175.195.$bl" A)
 echo "$bl: ${result:-OK}"
done

# Bash one-liner: addresses with the most bounces in the queue
postqueue -p | awk '/^[A-F0-9]/ {print $7}' | sort | uniq -c | sort -rn | head

# Stop auth attacks with Fail2ban (Postfix/Dovecot jail enabled)
sudo cat /etc/fail2ban/jail.d/mail.conf
# [postfix-sasl]
# enabled = true
# bantime = 24h

Mail Hosting Market in 2026: Decision Matrix

When comparing managed options to self-hosting, three axes matter: cost, operational burden, and trust boundary. Below is a rough positioning of regional providers and global hyperscalers (approximate prices, vary by provider, 2026 data).

• Regional mail hosting — Plans starting around $1–3 USD per user per month, often on local data centers (Turkey, EU, etc.). Typical: 10–50 GB mailbox, antispam included. Practical for small-to-medium teams.
• Dedicated corporate mail server (regional) — Around $30–200 USD per month. Unlimited accounts, 2–8 CPU, 40–100 GB SSD, local data residency. Makes sense for companies with 50+ users.
• Microsoft 365 / Google Workspace — $6–22 USD per user per month. Office tools + groupware + mail. Data lives outside your jurisdiction; privacy-law assessment may be required.
• Self-hosted (Postfix/Mailcow on a VPS) — Roughly $5–20 USD/month for VPS plus 4–6 hours/week of operations. More expensive than managed hosting at 3–10 users, cheaper at 100+ users.
• Hybrid (transactional ESP + mailbox host) — A dedicated ESP for campaigns/transactional mail, plus mail hosting for internal correspondence. The standard approach in large e-commerce and SaaS.

Among regional providers in Turkey, the well-known names include Natro, Turhost, Turkticaret.net, Doruk.Net, Teknosos, and Uzmanposta — each offers corporate email at different capacities and price points. When deciding, don’t look only at price; weigh local data-center presence, KVKK/GDPR compliance, antispam strength, IMAP push performance, mobile-device management panel, blacklist management, and the support SLA.

Email Security: Common Attacks

Email is still the most common entry point for corporate attacks. Verizon’s DBIR has put phishing in the top three for years. That’s why “designed to withstand attack scenarios” matters as much as the technical install of the mail server.

• Phishing & spear-phishing — Social engineering that earns trust through a familiar-looking sender. DMARC p=reject and user training are non-negotiable.
• BEC (Business Email Compromise) — CEO impersonation, fake invoices. According to FBI IC3 reports, losses exceed $50 billion USD annually.
• Spoofing — Forged mail in your domain’s name. SPF + DKIM + DMARC together are the only solution.
• Account takeover — Mailbox seizure with stolen credentials. Make 2FA (TOTP, WebAuthn) mandatory. Apply our password hashing guide.
• Malware attachments — Scan with ClamAV/Rspamd; don’t let .exe, .bat, .scr, or macro-laden .docm through.
• Open relay — A misconfigured Postfix must stay closed to outsiders; the order permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination is critical.
• SMTP smuggling (CVE-2023-51764) — Patched in Postfix and Sendmail in early 2024. Keep your server version up to date.

Backup Strategy

Email data is among an organization’s most sensitive digital assets; an unbacked mailbox is unacceptable. The 3-2-1 principle from our database backup strategies guide applies equally well to mailboxes.

• Mailbox snapshot — Run incremental backups of the Maildir directory daily with rsync --link-dest or borgbackup.
• Database dump — If Postfix/Dovecot stores transient state in MySQL/PostgreSQL, run mysqldump --single-transaction nightly.
• Off-site copy — At least one copy of every backup should live in a different geography or with a different provider.
• Restore drill — At least twice a year, restore a real mailbox into a clean environment.
• Retention policy — Define a deletion policy aligned with KVKK/GDPR retention windows (e.g., a departing employee’s mailbox stays readable for 90 days, archived for one year, then deleted).

Queue Management and Performance Tuning

On high-volume mail servers, queue management is the spine of performance. The Postfix queue is sensitive to disk I/O; it should run on SSD, and on multi-core servers default_process_limit should be tuned.

# /etc/postfix/main.cf - performance settings
default_process_limit = 100
smtp_destination_concurrency_limit = 20
smtp_destination_rate_delay = 1s # gentle per-recipient rate
smtp_extra_recipient_limit = 10

# Keep bounce and error queues aggressive
bounce_queue_lifetime = 2d
maximal_queue_lifetime = 5d
minimal_backoff_time = 300s
maximal_backoff_time = 4000s
queue_run_delay = 300s

# First the Postfix MTA side, then speed up the Dovecot mailbox side
# Maildir noindex (FastDir option)
# /etc/dovecot/conf.d/10-mail.conf: mail_fsync = always
# (always for backup integrity; 'optimized' for very-high-volume)

Mail Server Migration: Moving From an Old System

Migrating from one mail server to another is a multi-step operation involving DNS cutover, mailbox copying, and downtime management. imapsync is the de facto standard for per-user IMAP-to-IMAP copying.

# imapsync run for a single user
imapsync \
 --host1 old-mail.yourcompany.com --user1 alice@yourcompany.com --password1 'oldPassword' \
 --host2 mail.yourcompany.com --user2 alice@yourcompany.com --password2 'newPassword' \
 --ssl1 --ssl2 --no-foldersizes --automap

# Bulk run - users.txt: email;old_password;new_password
while IFS=';' read -r u p1 p2; do
 imapsync --host1 old.com --user1 "$u" --password1 "$p1" \
 --host2 new.com --user2 "$u" --password2 "$p2" \
 --ssl1 --ssl2 --automap
done < users.txt

# Method: lower the TTL first, prepare the second server, do the final sync, switch the MX

Migration-day order: (1) drop the old MX record’s TTL to 5 minutes and wait 24 hours so caches clear; (2) get the new server production-ready; (3) run the first imapsync pass; (4) flip the MX in DNS to the new server; (5) run a second, delta imapsync; (6) put the old server in read-only mode, wait 14 days, then shut it down.

Compliance: KVKK and GDPR

Email data counts as personal data and falls under KVKK and GDPR. Businesses operating in Turkey must know where their email data is stored, must rely on explicit consent or an adequacy decision when transferring it across borders, and must define written retention windows.

• Data residency — Is the server in Turkey, the EU, or the US? Decide based on KVKK rulings.
• Audit log — Mailbox accesses and admin actions must be logged. Stream them to a SIEM (Wazuh, ELK Stack) — see our ELK guide.
• Encryption at rest — Disk encryption with LUKS / dm-crypt; the mailbox directory should sit on an encrypted volume.
• Right to be forgotten — A process for deleting the mailbox plus backups within 30 days when a user requests it.
• DPIA — A Data Protection Impact Assessment for high-risk mail workloads (health, legal, finance).
• Subprocessor list — Document every third party that handles antispam, archive, or backup services.

Monitoring: Metrics and Alerts

“Silently broken” is the worst failure mode for a mail server — you don’t find out until users complain. Collect core metrics with Prometheus exporters and visualize them in Grafana. For details, read our Prometheus and Grafana guide.

• Postfix queue size — Use postfix_exporter; alert if it crosses 1,000 messages.
• Outbound bounce rate — Alarm above 5%.
• Inbound spam rate — A sudden drop signals broken Rspamd config.
• TLS handshake error rate — Useful for catching certificate-renewal failures.
• Mailbox disk fill — Warn at 85%, critical at 95%.
• Auth failure rate — A brute-force indicator; Fail2ban triggers on it.
• RBL listing alert — A script that queries your IP against 30 RBLs every 6 hours.

Mailcow: A Full Stack in 30 Minutes

Installing Postfix step by step is the best way to truly understand how it works. But if you need to ship to production fast, Mailcow brings up the entire stack (Postfix, Dovecot, Rspamd, ClamAV, SOGo, Nginx, Redis, MariaDB) in a few hours via a single docker-compose.

# Mailcow installation
sudo apt -y install docker.io docker-compose-plugin git
git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
sudo./generate_config.sh
# When asked 'Mail server hostname (FQDN)': mail.yourcompany.com

# After configuration
sudo docker compose pull
sudo docker compose up -d

# Admin panel
# https://mail.yourcompany.com -> admin / moohoo (change on first login)

# Backup
sudo./helper-scripts/backup_and_restore.sh backup all

# Update
sudo./update.sh

Hardware and Sizing for a Mail Server

The answer to “how much server do I need?” depends on user count, average mailbox size, and daily message volume. A general starting guideline:

• 1–25 users — 2 vCPU, 4 GB RAM, 80 GB SSD. A single VPS is enough.
• 25–100 users — 4 vCPU, 8 GB RAM, 200 GB SSD. SSD is ESSENTIAL — HDD-backed IMAP is painfully slow.
• 100–500 users — 8 vCPU, 16 GB RAM, 500 GB SSD; a separate backup volume.
• 500+ — MTA and store on separate servers; load balancer; replicated Dovecot store; likely an ESP integration too.
• RAM rule of thumb — Mailcow, for example, requires at least 6 GB; ClamAV alone uses 1.5–2 GB.
• Disk rule of thumb — Average 5 GB/user + log + index + backup multiplier (1.5x). 750 GB is safe for 100 users.

The Tightly-Packed 12-Item Checklist

After reading this guide, here is the most concise priority list to make sure your mail server is ready for everything:

• 1. PTR/rDNS configured, FCrDNS-compliant.
• 2. SPF record narrow (with -all), only your own servers.
• 3. DKIM 2048-bit, with an annual key-rotation plan.
• 4. DMARC at least p=quarantine, with RUA reports being monitored.
• 5. TLS 1.2+ enforced, certificates auto-renewing.
• 6. Submission port (587) closed behind STARTTLS + SASL; 25 open to the world but without auth (server-to-server) — relay must always stay locked down.
• 7. 2FA mandatory for mailbox users; admin panel restricted to an IP allowlist.
• 8. Rspamd + ClamAV active, with Bayes learning.
• 9. Daily backups, off-site, with a restore drill performed.
• 10. Eight core metrics monitored via Prometheus + Grafana.
• 11. Fail2ban Postfix + Dovecot jails active.
• 12. Postmaster Tools + DMARC reports reviewed weekly.

Frequently Asked Questions

How hard is it to set up a mail server?

Technically, you can stand up Postfix + Dovecot in 1–2 days. Operationally, building reputation, staying off blacklists, and keeping up with certificate and security updates demands ongoing effort. For teams that don’t want to manage all of this, managed mail hosting is a healthier choice.

Why do my emails go to spam, and which port should I use?

The most common causes: low reputation on a new IP/domain, missing SPF/DKIM/DMARC config, no PTR record, spam-triggering content, dirty recipient list, missing List-Unsubscribe header — start with mail-tester.com and push the score to 10/10. On port choice: 25 is for server-to-server transfer (MTA-to-MTA) and shouldn’t be used by user clients; 587 is for user submission with STARTTLS-encrypted credentials; 465 is implicit-TLS submission, brought back by RFC 8314. Modern clients accept both 587 and 465.

Will a self-hosted mail server work on a VPS, and which stack should I pick?

Yes, but pick a provider where port 25 is open — some keep it closed by default (you may need to file a request to open it). Common choices include Hetzner Cloud, Contabo, and OVH. Stay away from residential or dynamic IPs; building reputation from those is essentially impossible. Stack choice: Mailcow for the modern Docker approach, iRedMail for a clean install with classic Linux packages, or Postfix + Dovecot + Rspamd by hand to fully understand every component. From a production standpoint, all three can reach the same quality.

What should I look for when buying a domain mail server?

At minimum: local data-center presence (KVKK/GDPR convenience), per-user storage, IMAP push performance, antispam engine, SLA guarantee (99.9%+ uptime), automated DKIM/SPF/DMARC setup, mobile-device management panel, blacklist-management team, and 24/7 support in your language.

References and Further Reading

• RFC 5321 — SMTP
• RFC 5322 — Internet Message Format
• RFC 7208 — SPF
• RFC 6376 — DKIM
• RFC 7489 — DMARC
• Postfix documentation
• Dovecot documentation
• Rspamd docs
• Mailcow documentation
• mail-tester.com
• MXToolbox
• Google Postmaster Tools
• dmarc.org

Related Articles

• What Is DNS? Settings and Records Guide — MX, A, TXT records
• Free SSL With Let’s Encrypt — the foundation for mail TLS
• VPS Security Hardening — the Linux foundation for a mail server
• Fail2ban Guide — SMTP/IMAP brute-force protection
• Docker Compose — the foundation for stacks like Mailcow
• ELK Stack Log Analysis — monitoring mail logs
• Prometheus + Grafana — metrics via the postfix exporter