Performance Tuning¶

Optimizing Mailborder for maximum performance and efficiency.

Performance Assessment¶

Baseline Metrics¶

Establish baseline before tuning:

# Email processing rate
sudo mb-filter-stats --rate

# Service resource usage
systemctl status mb-rpcd mb-filter mb-milter

# Database performance
sudo mysql -e "SHOW STATUS LIKE 'Queries%'"

# Queue depth
postqueue -p | tail -n 1

Document baseline: - Emails processed per hour - Average processing time - CPU usage per service - Memory consumption - Database query rate - Queue depth

Identify Bottlenecks¶

CPU bottleneck:

top -b -n 1 | head -n 20
ps aux --sort=-%cpu | head -n 10

Memory bottleneck:

free -m
ps aux --sort=-%mem | head -n 10

Disk I/O bottleneck:

iostat -x 1 10
iotop -o

Network bottleneck:

iftop -i eth0
netstat -s | grep -i retrans

Database bottleneck:

sudo mysql -e "SHOW PROCESSLIST"
sudo mysql -e "SHOW STATUS LIKE 'Slow_queries'"

Service-Level Tuning¶

mb-rpcd Optimization¶

Increase worker processes:

sudo nano /etc/mailborder/mb-rpcd.conf

[rpcd]
# Increase from default 10
max_workers = 20

# Increase connection pool
db_pool_size = 20
db_pool_max = 50

# Adjust timeouts
request_timeout = 30
db_timeout = 10

# Enable connection reuse
connection_reuse = true

SystemD resource limits:

sudo systemctl edit mb-rpcd

[Service]
# Increase file descriptors
LimitNOFILE=65536

# Increase process limit
LimitNPROC=4096

# CPU and memory limits (adjust based on hardware)
CPUQuota=400%
MemoryMax=4G

Apply changes:

sudo systemctl daemon-reload
sudo systemctl restart mb-rpcd

mb-filter Optimization¶

Tune filter workers:

sudo nano /etc/mailborder/mb-filter.conf

[filter]
# Concurrent filter workers
workers = 8

# Maximum emails in processing queue
queue_size = 1000

# Processing timeouts
scan_timeout = 60
overall_timeout = 120

# ClamAV settings
clamav_max_filesize = 50M
clamav_max_scansize = 100M

# Rspamd settings
rspamd_timeout = 10
rspamd_workers = 4

Restart service:

sudo systemctl restart mb-filter

ClamAV Performance¶

Optimize ClamAV configuration:

sudo nano /etc/clamav/clamd.conf

# Increase threads (number of CPU cores)
MaxThreads = 4

# Reduce memory usage
MaxScanSize 100M
MaxFileSize 50M
MaxRecursion 5
MaxFiles 1000

# Disable unnecessary scanning
ScanPDF no
ScanSWF no
ScanXMLDOCS no

# Enable bytecode
Bytecode yes

# Stream scanning
StreamMaxLength 50M

Disable real-time signatures:

sudo nano /etc/clamav/freshclam.conf

# Use less frequent updates for better performance
Checks 12  # Every 2 hours instead of hourly

Restart ClamAV:

sudo systemctl restart clamd@scan

Rspamd Performance¶

Tune Rspamd workers:

sudo nano /etc/rspamd/local.d/worker-normal.inc

count = 4;  # Number of worker processes

max_tasks = 1000;  # Maximum tasks per worker

timeout = 10s;  # Scan timeout

Optimize fuzzy storage:

sudo nano /etc/rspamd/local.d/fuzzy_check.conf

rule "local" {
    algorithm = "mumhash";
    servers = "localhost:11335";

    # Reduce lookup overhead
    max_errors = 1;
    retransmits = 1;
    timeout = 2s;
}

Disable unused modules:

sudo nano /etc/rspamd/local.d/modules.conf

# Disable modules you don't use
modules {
    chartable {
        enabled = false;
    }
    mime_types {
        enabled = false;
    }
}

Restart Rspamd:

sudo systemctl restart rspamd

Database Tuning¶

MariaDB Configuration¶

Optimize MariaDB for performance:

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

[mysqld]
# Connection settings
max_connections = 200
max_connect_errors = 1000000
connect_timeout = 10
wait_timeout = 600
interactive_timeout = 600

# Buffer pool (set to 70-80% of available RAM)
innodb_buffer_pool_size = 4G
innodb_buffer_pool_instances = 4

# Log file size
innodb_log_file_size = 512M
innodb_log_buffer_size = 64M

# I/O settings
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000

# Thread settings
innodb_read_io_threads = 4
innodb_write_io_threads = 4
innodb_thread_concurrency = 8

# Query cache (deprecated in newer versions)
# query_cache_size = 0
# query_cache_type = 0

# Table cache
table_open_cache = 4000
table_definition_cache = 2000

# Sort and join buffers
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
join_buffer_size = 2M

# Temp tables
tmp_table_size = 64M
max_heap_table_size = 64M

# Slow query log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 2
log_queries_not_using_indexes = 0

Restart MariaDB:

sudo systemctl restart mysql

Database Optimization¶

Optimize tables:

sudo mysql mailborder << EOF
OPTIMIZE TABLE mb_log_audit;
OPTIMIZE TABLE mb_log_email;
OPTIMIZE TABLE mb_quarantine;
OPTIMIZE TABLE mb_users;
OPTIMIZE TABLE mb_auth_sessions;
EOF

Add indexes for common queries:

-- Index for log searches by date
CREATE INDEX idx_log_audit_created
ON mb_log_audit(created_at);

-- Index for email log searches
CREATE INDEX idx_log_email_received
ON mb_log_email(received_at);

-- Composite index for quarantine queries
CREATE INDEX idx_quarantine_recipient_date
ON mb_quarantine(recipient, quarantined_at);

-- Index for session lookups
CREATE INDEX idx_sessions_token
ON mb_auth_sessions(session_token);

Analyze tables:

sudo mysql mailborder << EOF
ANALYZE TABLE mb_log_audit;
ANALYZE TABLE mb_log_email;
ANALYZE TABLE mb_quarantine;
ANALYZE TABLE mb_users;
EOF

Redis Tuning¶

Optimize Redis configuration:

sudo nano /etc/redis/redis.conf

# Memory management
maxmemory 2gb
maxmemory-policy allkeys-lru

# Persistence (disable for pure cache)
save ""
appendonly no

# Network
tcp-backlog 511
timeout 300
tcp-keepalive 60

# Clients
maxclients 10000

# Performance
hz 10
dynamic-hz yes

# Latency
latency-monitor-threshold 100

Restart Redis:

sudo systemctl restart redis-server

Postfix Tuning¶

Optimize Postfix for high volume:

sudo nano /etc/postfix/main.cf

# Process limits
default_process_limit = 100
smtpd_client_connection_count_limit = 50
smtpd_client_connection_rate_limit = 100

# Concurrency
smtp_destination_concurrency_limit = 10
local_destination_concurrency_limit = 2

# Queue lifetime
maximal_queue_lifetime = 1h
bounce_queue_lifetime = 1h

# Delivery delays
minimal_backoff_time = 300s
maximal_backoff_time = 1800s

# Memory
header_size_limit = 102400
message_size_limit = 52428800

# TLS cache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

Reload Postfix:

sudo postfix reload

System-Level Tuning¶

Kernel Parameters¶

Optimize kernel settings:

sudo tee -a /etc/sysctl.conf << 'EOF'
# Network tuning
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 8192

# Connection tracking
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600

# File descriptors
fs.file-max = 2097152

# Shared memory
kernel.shmmax = 68719476736
kernel.shmall = 4294967296

# Swappiness
vm.swappiness = 10
EOF

sudo sysctl -p

File Descriptor Limits¶

Increase system-wide limits:

sudo nano /etc/security/limits.conf

* soft nofile 65536
* hard nofile 65536
mailborder soft nofile 65536
mailborder hard nofile 65536

Per-service limits:

# For each service
sudo systemctl edit mb-rpcd
sudo systemctl edit mb-filter
sudo systemctl edit nginx

[Service]
LimitNOFILE=65536

Filesystem Optimization¶

Mount options for performance:

sudo nano /etc/fstab

# Add noatime and nodiratime for better I/O performance
/dev/sda1 / ext4 defaults,noatime,nodiratime 0 1

# Separate mount for mail queue
/dev/sdb1 /var/spool/postfix ext4 defaults,noatime,data=writeback 0 2

Remount:

sudo mount -o remount /

PHP-FPM Tuning¶

Optimize PHP-FPM for web interface:

sudo nano /etc/php/8.2/fpm/pool.d/mailborder.conf

[mailborder]
; Process manager
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500

; Resource limits
php_admin_value[memory_limit] = 256M
php_admin_value[max_execution_time] = 60
php_admin_value[max_input_time] = 60

; OPcache settings
php_admin_value[opcache.enable] = 1
php_admin_value[opcache.memory_consumption] = 128
php_admin_value[opcache.interned_strings_buffer] = 8
php_admin_value[opcache.max_accelerated_files] = 10000
php_admin_value[opcache.validate_timestamps] = 0
php_admin_value[opcache.revalidate_freq] = 0

; Session handling
php_admin_value[session.save_handler] = redis
php_admin_value[session.save_path] = "tcp://127.0.0.1:6379"

Restart PHP-FPM:

sudo systemctl restart php8.2-fpm

Nginx Tuning¶

Optimize Nginx:

sudo nano /etc/nginx/nginx.conf

user www-data;
worker_processes auto;
worker_rlimit_nofile 65536;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # Basic settings
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    keepalive_requests 100;
    types_hash_max_size 2048;

    # Buffer sizes
    client_body_buffer_size 128k;
    client_max_body_size 50m;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 4k;

    # Timeouts
    client_body_timeout 12;
    client_header_timeout 12;
    send_timeout 10;

    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml text/javascript
               application/json application/javascript application/xml+rss;

    # FastCGI cache
    fastcgi_cache_path /var/cache/nginx/fastcgi
                       levels=1:2
                       keys_zone=mailborder:100m
                       inactive=60m
                       max_size=1g;
    fastcgi_cache_key "$scheme$request_method$host$request_uri";
    fastcgi_cache_use_stale error timeout invalid_header http_500;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;
}

Restart Nginx:

sudo systemctl restart nginx

Monitoring Performance Improvements¶

Before and After Comparison¶

Create performance test:

sudo tee /usr/local/bin/mb-perf-test.sh << 'EOF'
#!/bin/bash

echo "=== Mailborder Performance Test ==="
echo "Started: $(date)"
echo

# Send 100 test emails
START=$(date +%s)
for i in {1..100}; do
    echo "Test message $i" | mail -s "Performance Test $i" test@example.com
done

# Wait for queue to clear
while [ $(postqueue -p | tail -n 1 | awk '{print $5}') -gt 0 ]; do
    sleep 1
done

END=$(date +%s)
DURATION=$((END - START))

echo "Completed: $(date)"
echo "Duration: $DURATION seconds"
echo "Rate: $((100 / DURATION)) emails/second"
EOF

sudo chmod +x /usr/local/bin/mb-perf-test.sh

Run test:

sudo /usr/local/bin/mb-perf-test.sh

Continuous Monitoring¶

Setup performance monitoring:

# Install monitoring tools
sudo apt install sysstat prometheus-node-exporter

# Enable statistics collection
sudo systemctl enable sysstat
sudo systemctl start sysstat

Create monitoring dashboard:

watch -n 5 '
echo "=== Mailborder Performance ==="
echo "Queue: $(postqueue -p | tail -n 1)"
echo
echo "Services:"
systemctl status mb-rpcd --lines=0 | grep -E "Active|Memory|CPU"
systemctl status mb-filter --lines=0 | grep -E "Active|Memory|CPU"
echo
echo "Database:"
mysql -e "SHOW STATUS LIKE \"Threads_connected\""
echo
echo "Redis:"
redis-cli INFO stats | grep instantaneous_ops_per_sec
'

Performance Targets¶

Recommended targets:

Small Installation (< 1000 users): - Email processing: 100-500 emails/hour - Average processing time: < 100ms - Queue depth: < 50 messages - CPU usage: < 30% - Memory usage: < 2GB

Medium Installation (1000-10000 users): - Email processing: 500-5000 emails/hour - Average processing time: < 200ms - Queue depth: < 100 messages - CPU usage: < 50% - Memory usage: < 8GB

Large Installation (> 10000 users): - Email processing: > 5000 emails/hour - Average processing time: < 300ms - Queue depth: < 200 messages - CPU usage: < 70% - Memory usage: < 16GB

Hardware Recommendations¶

Minimum Requirements¶

Small installation: - CPU: 2 cores - RAM: 4GB - Disk: 50GB SSD - Network: 100Mbps

Medium installation: - CPU: 4 cores - RAM: 8GB - Disk: 100GB SSD - Network: 1Gbps

Large installation: - CPU: 8+ cores - RAM: 16GB+ - Disk: 200GB+ SSD (RAID 10) - Network: 1Gbps+ (bonded)

Scaling Up vs Out¶

Vertical Scaling (Scale Up): - Add more CPU cores - Increase RAM - Upgrade to faster SSD - Simple, no architectural changes - Limited by hardware maximums

Horizontal Scaling (Scale Out): - Add more servers - Implement load balancing - Use shared storage and database - More complex but unlimited scaling - See Clustering

Troubleshooting Performance Issues¶

Slow email processing: 1. Check ClamAV signature database size 2. Verify database indexes exist 3. Review slow query log 4. Check for disk I/O bottlenecks 5. Increase filter workers

High CPU usage: 1. Identify top CPU consumers 2. Reduce ClamAV recursion depth 3. Optimize Rspamd rules 4. Disable unused scanning features 5. Add more workers

High memory usage: 1. Reduce buffer pool sizes 2. Limit concurrent processes 3. Disable query cache (if enabled) 4. Reduce ClamAV MaxThreads 5. Add more RAM or enable swap

High disk I/O: 1. Move queue to separate disk 2. Use SSD for database 3. Disable excessive logging 4. Optimize log rotation 5. Archive old data