🛡️ Ubuntu Server Hardening and Optimization Guide

A Comprehensive Guide to Securing and Optimizing Your Ubuntu Server

📋 Table of Contents

⚙️ Configuring Swappiness and Cache Pressure

Before making changes to kernel parameters, it's essential to understand what swappiness and cache pressure control on your Linux system.

What is Swappiness?
Swappiness is a kernel parameter that controls how aggressively the system swaps memory pages from RAM to disk. The value ranges from 0 to 100, where a lower value means less swapping.

Step 1: Check Current Settings

First, let's examine the current kernel parameters using the sysctl command:

sudo sysctl -a

This command displays all kernel parameters. Look for these two important values:

Parameter Default Value Recommended Value Purpose
vm.swappiness 60 1 Controls memory swapping behavior
vm.vfs_cache_pressure 100 50 Controls cache reclamation tendency

Step 2: Create Override Configuration File

Navigate to the sysctl configuration directory and create a custom override file:

cd /etc/sysctl.d/
sudo nano custom_overrides.conf
Example Configuration File Content:
# Swappiness and cache pressure settings
vm.swappiness=1
vm.vfs_cache_pressure=50

Configuration Flow Diagram

Check Current
Settings
sysctl -a
→
Create Override
File
custom_overrides.conf
→
Add
Directives
vm.swappiness=1
→
Reboot
Server
sudo reboot

Step 3: Apply Changes

Save the file and reboot the server to apply the changes:

sudo reboot

Step 4: Verify Configuration

After rebooting, verify that the changes have been applied:

sudo sysctl -a | grep swappiness
sudo sysctl -a | grep cache_pressure
✓ Expected Output:
vm.swappiness = 1
vm.vfs_cache_pressure = 50
⚠️ Important Note: If you upgrade your server's RAM, you'll need to adjust the swap file size accordingly. Cache pressure might also benefit from adjustments with additional RAM - a higher value (above 50) could further improve performance.

🔒 Hardening Shared Memory

Shared memory is an inter-process communication (IPC) mechanism that allows multiple processes to access the same memory area. While efficient, it poses security risks if not properly secured.

Why Harden Shared Memory?
If a malicious user gains access to shared memory, they can potentially access sensitive data or modify data that other processes rely on. Implementing access controls is crucial for security.

Understanding Security Parameters

Parameter Description Security Benefit
noexec Prevents execution of binaries Blocks malicious code execution
nosuid Blocks SUID and SGID bits Prevents privilege escalation
nodev Prevents device file usage Restricts device access
Understanding SUID and SGID:
SUID (Set User ID) and SGID (Set Group ID) are special permissions that change how files are executed. Normally, a program runs with the permissions of the user who executes it. With SUID, the program runs with the owner's permissions, even if launched by another user. While useful for granting temporary privileges, they should be used cautiously due to security risks.

Implementation Steps

1Create a backup of the fstab file:

sudo cp /etc/fstab /etc/fstab.backup

2Edit the fstab file:

sudo nano /etc/fstab

3Add the following entries below the swap file section:

# Harden shared memory
tmpfs /run/shm tmpfs defaults,noexec,nosuid,nodev 0 0

4Reboot the server:

sudo reboot

5Verify the hardening:

mount | grep shm

Shared Memory Hardening Process

Backup
fstab file
→
Edit
/etc/fstab
→
Add Security
Parameters
→
Reboot &
Verify

🚫 Disabling IPv6

While IPv6 offers many benefits, it can introduce complexity and potential security vulnerabilities if not properly managed. Here are key considerations:

Reasons to Disable IPv6

  • Increased Complexity: Managing both IPv4 and IPv6 configurations increases network administration complexity
  • Security Risks: IPv6 introduces new attack vectors that may not be covered by existing security tools
  • Compatibility Issues: Older hardware and software may not fully support IPv6
  • Performance Concerns: Unoptimized IPv6 infrastructure can lead to slower network speeds
  • Troubleshooting Challenges: Dual-stack configurations can make issue diagnosis more difficult

Checking IPv6 Status

First, verify if IPv6 is currently enabled:

ip a | grep inet6
Note: If the command returns no output, IPv6 is already disabled. If you see "inet6" addresses, IPv6 is currently enabled.

Disabling IPv6 Using GRUB

1Navigate to the GRUB configuration directory:

cd /etc/default

2Open the GRUB configuration file:

sudo nano grub

3Find the line containing GRUB_CMDLINE_LINUX and add the following between the quotation marks:

ipv6.disable=1
Example:
GRUB_CMDLINE_LINUX="ipv6.disable=1"

4Update the GRUB bootloader:

sudo update-grub

5Reboot the server:

sudo reboot

6Verify IPv6 is disabled:

ip a | grep inet6
✓ Success: If no output is returned, IPv6 has been successfully disabled.

IPv6 Disabling Workflow

Check IPv6
Status
→
Edit GRUB
Configuration
→
Update
Bootloader
→
Reboot &
Verify

🌐 Network Layer Hardening and Optimization

Hardening and optimizing the network layer improves both security and performance. This involves configuring multiple kernel parameters to protect against attacks and handle network traffic efficiently.

Optimization Parameters

Parameter Purpose Benefit
net.ipv4.ip_local_port_range Expands available port range More concurrent connections
fs.file-max Increases maximum open files Better handle multiple connections
fs.suid_dumpable Disables core dumps for SUID Enhanced security
net.core.somaxconn Increases connection queue Handle more connection requests
net.core.netdev_max_backlog Increases packet queue size Better handle traffic bursts

Security Hardening Measures

Key Security Protections:
  • IP Spoofing Prevention: Blocks packets with false source addresses
  • SYN Flood Protection: Defends against denial-of-service attacks
  • Source Routing Disable: Prevents malicious traffic redirection
  • Exec Shield: Provides additional memory protection

Configuration Steps

1Navigate to the sysctl configuration directory:

cd /etc/sysctl.d/

2Edit the custom overrides file:

sudo nano custom_overrides.conf

3Add network hardening directives:

IP Spoofing Protection:
# IP Spoofing protection
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
SYN Flood Protection:
# SYN flood protection
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_syn_backlog=2048
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_syn_retries=5
Disable Source Routing:
# Disable source routing
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.default.accept_source_route=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.default.accept_source_route=0

4Add network optimization directives:

Performance Optimization:
# Increase number of usable ports
net.ipv4.ip_local_port_range=1024 65535

# Increase file handles and inode cache
fs.file-max=2097152

# Restrict core dumps
fs.suid_dumpable=0

# Increase incoming connections
net.core.somaxconn=65535
net.core.netdev_max_backlog=5000

# Increase memory buffers
net.core.optmem_max=25165824

# Configure send/receive buffers
net.core.rmem_default=31457280
net.core.rmem_max=67108864
net.core.wmem_default=31457280
net.core.wmem_max=67108864
⚠️ Important: Increase buffer values gradually. While higher values improve performance, they also increase memory usage. Monitor your server's memory consumption and adjust accordingly.

5Save the file and reboot:

sudo reboot

Network Security Layers

Layer 1
IP Spoofing
Protection
→
Layer 2
SYN Flood
Defense
→
Layer 3
Source Routing
Disabled
→
Layer 4
Performance
Optimization

Understanding SYN Flood Attacks

A SYN flood is a form of denial-of-service attack where an attacker rapidly initiates connections without completing them. The server spends resources waiting for half-open connections, potentially making it unresponsive to legitimate traffic. Our configuration protects against this by enabling SYN cookies and limiting retry attempts.

✓ Server Stability Note: During initial setup, frequent reboots are necessary to apply configurations. However, one of the main properties of a server distribution is stability. After completing the initial configuration, your server should require minimal reboots, ensuring maximum uptime for your services.

📝 Summary

By following this comprehensive guide, you have successfully:

  • Optimized memory management through swappiness and cache pressure configuration
  • Hardened shared memory to prevent unauthorized access
  • Disabled IPv6 to reduce complexity and potential attack vectors
  • Implemented robust network layer security measures
  • Optimized network performance for better throughput and connection handling
Best Practices:
  • Always create backups before modifying system files
  • Document all changes with comments in configuration files
  • Test configurations in a development environment first
  • Monitor system performance after applying changes
  • Keep your server updated with security patches
  • Regularly review and adjust parameters based on workload

Your Ubuntu server is now hardened against common attack vectors and optimized for better performance. Regular maintenance and monitoring will ensure continued security and efficiency.