What Is a Mail Server? Setup, SMTP/IMAP and Domain

A mail server is the mostly invisible but mission-critical software stack that makes email work at internet scale. Every time a user clicks “send,” dozens of processes, certificates, DNS lookups, and protocol exchanges land on the mail server’s shoulders. This guide doesn’t stop at conceptual definitions; it walks you through how to build your own mail server from scratch, which DNS records you must publish, how to protect deliverability, and the real-world difference between buying managed mail hosting and running the server under your own control — all backed by 2026 data.

What Is a Mail Server? A Definition From the Roots Up

A mail server is the collection of programs that accept, relay, store, and deliver email messages over the internet. There is no single piece of software at its core; a mail server is actually an architecture of at least three distinct components: the MTA (Mail Transfer Agent — moves mail from server to server), the MDA (Mail Delivery Agent — writes incoming messages into the right user’s mailbox), and the MUA (Mail User Agent — the user-facing client like Outlook, Thunderbird, or webmail).

The backbone underneath this architecture is a handful of RFCs published since 1982: RFC 5321 (SMTP), RFC 5322 (Internet Message Format), RFC 3501 (IMAP4rev1). SMTP’s original design goal was store-and-forward; it still operates on that same fundamental logic today — but a dozen layers of TLS, authentication, antispam, and DKIM have been stacked on top.

How a Mail Server Works: The Journey of an Email

Most users have no idea what unfolds behind the scenes when they hit send. In reality, every message passes through at least six checkpoints in roughly 200 milliseconds to 30 seconds. From left to right: sender MUA, sender submission server (port 587), sender outbound MTA, recipient domain’s MX server (port 25), recipient IMAP/POP3 store, and finally the recipient MUA. If any link in that chain breaks, the message either waits, bounces back, or lands in the spam folder.

1. Submission — After authenticating the user, the MUA hands the message to the submission server on port 587.
2. Outbound queue — The MTA queues the message and runs an MX DNS lookup for the recipient domain.
3. SMTP conversation — The sending MTA connects to the recipient MX on port 25 and delivers the message via EHLO, MAIL FROM, RCPT TO, and DATA.
4. Antispam & auth check — The receiving server runs SPF, DKIM, DMARC, RBL, and content-scoring tests.
5. Local delivery — The MDA writes the message into the user’s mailbox (Maildir or mbox format).
6. Retrieval — The user pulls the message via IMAP or POP3, or reads it through webmail directly on the server.

SMTP, IMAP, and POP3: Which Protocol Does What?

Mixing up the three email protocols is a common mistake. The short version: SMTP is for sending; IMAP and POP3 are for receiving. SMTP works on a push model, IMAP/POP3 on a pull model.

SMTP (Simple Mail Transfer Protocol, RFC 5321) — Carries messages from one server to another. Submission uses port 587 (STARTTLS), server-to-server transfer uses port 25, and legacy implicit TLS uses 465.
IMAP (Internet Message Access Protocol, RFC 3501) — Keeps messages on the server, syncs folder structure and read/unread flags. The default choice for accessing mail from multiple devices. Port 143 (STARTTLS) or 993 (TLS).
POP3 (Post Office Protocol v3, RFC 1939) — Downloads messages from the server, usually deleting them in the process. Still useful for offline-archive scenarios. Port 110 (STARTTLS) or 995 (TLS).
Always use submission — ISPs commonly block port 25. For Outlook/Thunderbird, configure 587 + STARTTLS without exception.
Implicit TLS vs STARTTLS — Implicit TLS encrypts the connection from the moment it opens; STARTTLS starts in plain text and upgrades to encryption. The practical difference is small in modern clients, but auditors prefer implicit TLS.

Domain Mail Server: A Mail System That Runs on Your Own Domain

A “domain mail server” — commonly called “branded email” — lets you operate addresses in the form mailbox@yourcompany.com. The difference between info@gmail.com and info@yourcompany.com isn’t just aesthetic; the latter signals trust, professionalism, and brand consistency, and measurably increases customer response rates.

Three building blocks are essential to set up a domain mail server: a domain name (see our domain and WHOIS guide), the correct DNS records (MX, SPF, DKIM, DMARC), and either a self-hosted mail server or a managed mail hosting product. If there’s a gap between domain registration and mail setup, expect 4–48 hours for full DNS propagation.

The DNS Side: MX, A, SPF, DKIM, and DMARC Records

No matter how well-tuned your mail server is, if the DNS records are wrong your messages will either never arrive or end up in spam. You must know these five record types by heart: MX (the recipient server’s address), A/AAAA (the IP of your mail host), SPF (TXT — who is allowed to send on your behalf), DKIM (TXT — cryptographically signs messages), and DMARC (TXT — defines what to do when SPF/DKIM fails).

In SPF, the trailing -all means “hard fail” — messages from IPs not on the list are rejected. Start with ~all (soft fail) in a test environment, then tighten to -all once you’re confident your records are right. The DKIM selector name (mail in the example) must match the key pair generated on the server. For DMARC, it’s a good idea to begin with p=none (report only, take no action); after 1–2 weeks of data, move to p=quarantine, then to p=reject.

Reverse DNS (PTR), HELO, and FCrDNS

Most spam filters dock points if your mail server has no PTR (reverse DNS) record, or if its HELO banner doesn’t match the PTR. PTR records are typically created in your IP provider’s panel; most VPS providers offer a “Reverse DNS” or “rDNS” field in the user interface. Forward-confirmed reverse DNS (FCrDNS) means the A record for mail.yourcompany.com resolves to x.y.z.w and that same IP’s PTR points back to mail.yourcompany.com. If both directions don’t agree, Gmail, Outlook, and Yahoo will silently filter your mail.

Setting Up a Mail Server: Two Paths

Behind the search query “how to set up a mail server” sit two completely different intents: some people want “buy ready-made, just use it”; others want “I’m going to roll up my sleeves and run my own server.” Both are legitimate choices, but the costs and operational burden differ.

Managed mail hosting / corporate email plan — You rent per-user domain mailboxes from a multi-tenant server operated by a provider. DNS, antispam, backups, updates, and blacklist management are the provider’s responsibility.
Self-hosted mail server — You install Postfix + Dovecot, iRedMail, Mailcow, or Mail-in-a-Box on your own VPS and run all of the infrastructure yourself. You get full control, but the operational load is real.

For small teams, self-hosting is a romantic but risky choice: blacklist avoidance, IP reputation, and antispam tuning add up to roughly 4–6 hours of weekly maintenance on their own. Building IP reputation today can take years; one compromised account can halt your outbound mail for weeks. Our recommendation: teams smaller than 50 should use managed mail hosting and configure SPF/DKIM/DMARC correctly on top; larger teams or those with regulatory obligations should move to self-hosting.

The Self-Hosted Stack: What Will You Install?

The open-source mail-server ecosystem is mature. As of 2026, the most common combinations are:

Postfix + Dovecot + Rspamd + OpenDKIM — The classic “build from components” approach. Maximum flexibility, minimum magic. Best support for debugging and performance tuning.
iRedMail — Installs Postfix + Dovecot + Amavis + SpamAssassin with a single script. Supports CentOS/Ubuntu/Debian, with a free Community edition. Very practical for small-to-medium scale.
Mailcow — A dockerized mail server. Brings up Postfix, Dovecot, Rspamd, ClamAV, SOGo (groupware), Nginx, Redis, and MariaDB with a single docker-compose. The standout choice for modern teams. Read our Docker Compose guide for context.
Mail-in-a-Box — Promises “a mail server with one command.” Better suited to individual or small-scale use. Customization is hard.
Modoboa — A Python-based admin panel integrated with Postfix/Dovecot/Rspamd. Attractive for teams that want a simple web UI.
Zimbra Open Source / Carbonio — Groupware-focused, bundling calendar, contacts, and file sharing. An open-source alternative to Exchange.

Step 1: Server Prep and Hardening

For the rest of this guide we’ll walk through the most common and transparent approach — Postfix + Dovecot + Rspamd + OpenDKIM — on Ubuntu 24.04 LTS, step by step. Before installing the mail server, lay down the security basics on your VPS; for a deeper dive, read our VPS security hardening and Fail2ban guide. Mail servers attract some of the highest brute-force traffic on the internet; 50–100 failed SSH login attempts per minute is normal.

Step 2: Installing the TLS Certificate

A modern mail server must run with TLS. Let’s Encrypt provides free, auto-renewing certificates and is the most common choice for mail servers. For step-by-step configuration, see our Let’s Encrypt guide.

Step 3: Installing and Configuring Postfix

Postfix, written by Wietse Venema in 1998 as a Sendmail alternative, is the most widely deployed MTA on the internet today. Its modular design, auditable configuration, and strong security record make it a fit for everything from a small office server to ISP-scale infrastructure.

In /etc/postfix/master.cf, enable the submission (587) and smtps (465) ports; both should always require SASL. Keep postfix.org/postconf.5 close at hand as a reference for every Postfix directive.

Step 4: Installing Dovecot (IMAP/POP3 + LMTP)

Dovecot takes on the MDA + IMAP/POP3 role where Postfix is the MTA. It receives messages from Postfix over LMTP, writes them to the user’s Maildir, and serves them over IMAP/POP3. Postfix also leverages Dovecot’s auth service.

Step 5: DKIM Signing With OpenDKIM

DKIM applies a cryptographic signature to every outbound message; the receiving server fetches the public key from DNS and verifies the signature. It’s stronger than SPF because it also proves the message body wasn’t altered. OpenDKIM is the most widely used implementation that talks to Postfix as a milter.

As of 2026, the minimum acceptable DKIM key size is 2048-bit RSA; 1024-bit should only appear on legacy systems. Best practice is to rotate the key once a year — name the selector mail2026, then mail2027 the following year.

Step 6: Antispam With Rspamd

Rspamd is the modern successor to the older SpamAssassin: written in C, dramatically faster, and bundling Bayes/neural-network learning, RBL checks, greylisting, rate-limiting, and hundreds of other rules. It integrates with Postfix as a milter.

Redis is the data store Rspamd uses for Bayes data, fuzzy hashes, and rate limiting. For general Redis usage, see our Redis basics guide. Rspamd learns automatically from messages users move into the “spam” folder — you can wire up that training pipeline through IMAP Sieve filters.

Step 7: Webmail Setup (Roundcube)

Your users will need a webmail interface to read mail in the browser. The most common options are Roundcube (PHP), SnappyMail (PHP, lightweight), RainLoop (no longer maintained), and SOGo (groupware). Serving it through Nginx is the default step in most deployments — refer to our Nginx configuration guide.

Testing and Validation

After the mail server is built, you absolutely must test it. Before going live with a real user, every item on this checklist needs to pass:

Send a test message to mail-tester.com — aim for 10/10.
Validate MX, SPF, DKIM, and DMARC records via mxtoolbox.com.
Inspect your SMTP TLS handshake with checktls.com.
Send test messages to Gmail, Outlook.com, Yahoo, and Yandex accounts and check the inbox/spam split.
Test from the CLI with swaks --to test@gmail.com --from postmaster@yourcompany.com --server localhost.
Open the message in Gmail, click “Show original,” and confirm SPF=PASS, DKIM=PASS, DMARC=PASS.
Confirm DMARC reports start arriving at dmarc@yourcompany.com (1–3 days).

Deliverability: The Invisible Battle

Even if your mail server is technically correct, your messages can still land in spam. The cause isn’t technical — it’s reputation. New IPs and new domains are treated with suspicion by Gmail, Microsoft 365, and Yahoo; building reputation takes weeks.

IP warm-up — Send 50–100 messages/day in the first days from a new server; double the volume each week.
List hygiene — Remove hard bounces from the list immediately. A bounce rate above 5% will dent reputation.
One-click unsubscribe (List-Unsubscribe header) — Add RFC 8058 / RFC 2369 compliant List-Unsubscribe: <mailto:...>, <https:...> and List-Unsubscribe-Post: List-Unsubscribe=One-Click headers. Gmail/Yahoo have required this since 2024.
Complaint rate (FBL feedback loop) — Keep it below 0.3%.
Stream separation — Don’t only send marketing; route transactional and marketing traffic through different IPs/domains.
BIMI — Combine DMARC p=quarantine+ with a BIMI record; your logo will appear next to the sender name in Gmail/Yahoo.
Postmaster Tools — Register your domain at postmaster.google.com and postmaster.live.com to view reputation, spam rate, and auth data.

Blacklist (DNSBL) Monitoring

If your IP lands on Spamhaus ZEN, SORBS, Barracuda, UCEPROTECT, or Backscatterer, your messages start being rejected immediately. Compromised accounts, virus-infected endpoints, and poorly configured contact forms are common reasons for getting listed.

Mail Hosting Market in 2026: Decision Matrix

When comparing managed options to self-hosting, three axes matter: cost, operational burden, and trust boundary. Below is a rough positioning of regional providers and global hyperscalers (approximate prices, vary by provider, 2026 data).

Regional mail hosting — Plans starting around $1–3 USD per user per month, often on local data centers (Turkey, EU, etc.). Typical: 10–50 GB mailbox, antispam included. Practical for small-to-medium teams.
Dedicated corporate mail server (regional) — Around $30–200 USD per month. Unlimited accounts, 2–8 CPU, 40–100 GB SSD, local data residency. Makes sense for companies with 50+ users.
Microsoft 365 / Google Workspace — $6–22 USD per user per month. Office tools + groupware + mail. Data lives outside your jurisdiction; privacy-law assessment may be required.
Self-hosted (Postfix/Mailcow on a VPS) — Roughly $5–20 USD/month for VPS plus 4–6 hours/week of operations. More expensive than managed hosting at 3–10 users, cheaper at 100+ users.
Hybrid (transactional ESP + mailbox host) — A dedicated ESP for campaigns/transactional mail, plus mail hosting for internal correspondence. The standard approach in large e-commerce and SaaS.

Among regional providers in Turkey, the well-known names include Natro, Turhost, Turkticaret.net, Doruk.Net, Teknosos, and Uzmanposta — each offers corporate email at different capacities and price points. When deciding, don’t look only at price; weigh local data-center presence, KVKK/GDPR compliance, antispam strength, IMAP push performance, mobile-device management panel, blacklist management, and the support SLA.

Email Security: Common Attacks

Email is still the most common entry point for corporate attacks. Verizon’s DBIR has put phishing in the top three for years. That’s why “designed to withstand attack scenarios” matters as much as the technical install of the mail server.

Phishing & spear-phishing — Social engineering that earns trust through a familiar-looking sender. DMARC p=reject and user training are non-negotiable.
BEC (Business Email Compromise) — CEO impersonation, fake invoices. According to FBI IC3 reports, losses exceed $50 billion USD annually.
Spoofing — Forged mail in your domain’s name. SPF + DKIM + DMARC together are the only solution.
Account takeover — Mailbox seizure with stolen credentials. Make 2FA (TOTP, WebAuthn) mandatory. Apply our password hashing guide.
Malware attachments — Scan with ClamAV/Rspamd; don’t let .exe, .bat, .scr, or macro-laden .docm through.
Open relay — A misconfigured Postfix must stay closed to outsiders; the order permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination is critical.
SMTP smuggling (CVE-2023-51764) — Patched in Postfix and Sendmail in early 2024. Keep your server version up to date.

Backup Strategy

Email data is among an organization’s most sensitive digital assets; an unbacked mailbox is unacceptable. The 3-2-1 principle from our database backup strategies guide applies equally well to mailboxes.

Mailbox snapshot — Run incremental backups of the Maildir directory daily with rsync --link-dest or borgbackup.
Database dump — If Postfix/Dovecot stores transient state in MySQL/PostgreSQL, run mysqldump --single-transaction nightly.
Off-site copy — At least one copy of every backup should live in a different geography or with a different provider.
Restore drill — At least twice a year, restore a real mailbox into a clean environment.
Retention policy — Define a deletion policy aligned with KVKK/GDPR retention windows (e.g., a departing employee’s mailbox stays readable for 90 days, archived for one year, then deleted).

Queue Management and Performance Tuning

On high-volume mail servers, queue management is the spine of performance. The Postfix queue is sensitive to disk I/O; it should run on SSD, and on multi-core servers default_process_limit should be tuned.

Mail Server Migration: Moving From an Old System

Migrating from one mail server to another is a multi-step operation involving DNS cutover, mailbox copying, and downtime management. imapsync is the de facto standard for per-user IMAP-to-IMAP copying.

Migration-day order: (1) drop the old MX record’s TTL to 5 minutes and wait 24 hours so caches clear; (2) get the new server production-ready; (3) run the first imapsync pass; (4) flip the MX in DNS to the new server; (5) run a second, delta imapsync; (6) put the old server in read-only mode, wait 14 days, then shut it down.

Email data counts as personal data and falls under KVKK and GDPR. Businesses operating in Turkey must know where their email data is stored, must rely on explicit consent or an adequacy decision when transferring it across borders, and must define written retention windows.

Data residency — Is the server in Turkey, the EU, or the US? Decide based on KVKK rulings.
Audit log — Mailbox accesses and admin actions must be logged. Stream them to a SIEM (Wazuh, ELK Stack) — see our ELK guide.
Encryption at rest — Disk encryption with LUKS / dm-crypt; the mailbox directory should sit on an encrypted volume.
Right to be forgotten — A process for deleting the mailbox plus backups within 30 days when a user requests it.
DPIA — A Data Protection Impact Assessment for high-risk mail workloads (health, legal, finance).
Subprocessor list — Document every third party that handles antispam, archive, or backup services.

Monitoring: Metrics and Alerts

“Silently broken” is the worst failure mode for a mail server — you don’t find out until users complain. Collect core metrics with Prometheus exporters and visualize them in Grafana. For details, read our Prometheus and Grafana guide.

Postfix queue size — Use postfix_exporter; alert if it crosses 1,000 messages.
Outbound bounce rate — Alarm above 5%.
Inbound spam rate — A sudden drop signals broken Rspamd config.
TLS handshake error rate — Useful for catching certificate-renewal failures.
Mailbox disk fill — Warn at 85%, critical at 95%.
Auth failure rate — A brute-force indicator; Fail2ban triggers on it.
RBL listing alert — A script that queries your IP against 30 RBLs every 6 hours.

Mailcow: A Full Stack in 30 Minutes

Installing Postfix step by step is the best way to truly understand how it works. But if you need to ship to production fast, Mailcow brings up the entire stack (Postfix, Dovecot, Rspamd, ClamAV, SOGo, Nginx, Redis, MariaDB) in a few hours via a single docker-compose.

Hardware and Sizing for a Mail Server

The answer to “how much server do I need?” depends on user count, average mailbox size, and daily message volume. A general starting guideline:

1–25 users — 2 vCPU, 4 GB RAM, 80 GB SSD. A single VPS is enough.
25–100 users — 4 vCPU, 8 GB RAM, 200 GB SSD. SSD is ESSENTIAL — HDD-backed IMAP is painfully slow.
100–500 users — 8 vCPU, 16 GB RAM, 500 GB SSD; a separate backup volume.
500+ — MTA and store on separate servers; load balancer; replicated Dovecot store; likely an ESP integration too.
RAM rule of thumb — Mailcow, for example, requires at least 6 GB; ClamAV alone uses 1.5–2 GB.
Disk rule of thumb — Average 5 GB/user + log + index + backup multiplier (1.5x). 750 GB is safe for 100 users.

The Tightly-Packed 12-Item Checklist

After reading this guide, here is the most concise priority list to make sure your mail server is ready for everything:

1. PTR/rDNS configured, FCrDNS-compliant.
2. SPF record narrow (with -all), only your own servers.
3. DKIM 2048-bit, with an annual key-rotation plan.
4. DMARC at least p=quarantine, with RUA reports being monitored.
5. TLS 1.2+ enforced, certificates auto-renewing.
6. Submission port (587) closed behind STARTTLS + SASL; 25 open to the world but without auth (server-to-server) — relay must always stay locked down.
7. 2FA mandatory for mailbox users; admin panel restricted to an IP allowlist.
8. Rspamd + ClamAV active, with Bayes learning.
9. Daily backups, off-site, with a restore drill performed.
10. Eight core metrics monitored via Prometheus + Grafana.
11. Fail2ban Postfix + Dovecot jails active.
12. Postmaster Tools + DMARC reports reviewed weekly.

Frequently Asked Questions

How hard is it to set up a mail server?

Technically, you can stand up Postfix + Dovecot in 1–2 days. Operationally, building reputation, staying off blacklists, and keeping up with certificate and security updates demands ongoing effort. For teams that don’t want to manage all of this, managed mail hosting is a healthier choice.

Why do my emails go to spam, and which port should I use?

The most common causes: low reputation on a new IP/domain, missing SPF/DKIM/DMARC config, no PTR record, spam-triggering content, dirty recipient list, missing List-Unsubscribe header — start with mail-tester.com and push the score to 10/10. On port choice: 25 is for server-to-server transfer (MTA-to-MTA) and shouldn’t be used by user clients; 587 is for user submission with STARTTLS-encrypted credentials; 465 is implicit-TLS submission, brought back by RFC 8314. Modern clients accept both 587 and 465.

Will a self-hosted mail server work on a VPS, and which stack should I pick?

Yes, but pick a provider where port 25 is open — some keep it closed by default (you may need to file a request to open it). Common choices include Hetzner Cloud, Contabo, and OVH. Stay away from residential or dynamic IPs; building reputation from those is essentially impossible. Stack choice: Mailcow for the modern Docker approach, iRedMail for a clean install with classic Linux packages, or Postfix + Dovecot + Rspamd by hand to fully understand every component. From a production standpoint, all three can reach the same quality.

What should I look for when buying a domain mail server?

At minimum: local data-center presence (KVKK/GDPR convenience), per-user storage, IMAP push performance, antispam engine, SLA guarantee (99.9%+ uptime), automated DKIM/SPF/DMARC setup, mobile-device management panel, blacklist-management team, and 24/7 support in your language.

References and Further Reading

What Is DNS? Settings and Records Guide — MX, A, TXT records
Free SSL With Let’s Encrypt — the foundation for mail TLS
VPS Security Hardening — the Linux foundation for a mail server
Fail2ban Guide — SMTP/IMAP brute-force protection
Docker Compose — the foundation for stacks like Mailcow
ELK Stack Log Analysis — monitoring mail logs
Prometheus + Grafana — metrics via the postfix exporter

Get expert support for your mail server setup

For Postfix, Dovecot, Mailcow, or corporate mail-hosting selection; SPF/DKIM/DMARC configuration, deliverability optimization, and KVKK-compliant infrastructure, work with our team to get in touch

Readers of this article also read these

hosting 34 min

Software Development

SEO & Optimization

API & Integration

Web Hosting

VPS Server

Dedicated Server

Domain & SSL

Why KEYDAL?

Projelerimiz

About Us

Contact