Server Monitoring and Maintenance Guide

Professional Guide for Ubuntu Server Administration, Monitoring, and Security

Table of Contents

1. Server Monitoring Solutions

External Monitoring: Uptime Robot

Uptime Robot offers an excellent free service to monitor your site's availability and uptime. It provides a way to ensure that your web resources are up and running smoothly. The features offered in the Uptime Robot Free plan should be adequate for most sites. It's actually a great way to monitor the uptime of your sites without requiring any server-side agent installation.

Key Features: External monitoring, no server resources required, free tier available, real-time alerts

Server Resource Monitoring: Htop

Htop is an excellent tool for monitoring server resources and processes in real time. It provides a comprehensive overview of CPU, memory, and swap usage, allowing users to quickly identify any resource-intensive processes. Htop makes it easy to pinpoint problematic processes by displaying detailed information such as CPU and memory usage for each process.

Installing and Using Htop

sudo apt install htop

Running Htop

htop
Htop Keyboard Shortcuts:
F6 - Sort processes by various criteria (CPU, Memory, etc.)
ESC - Return to main view
Q - Quit Htop

Htop Workflow

Launch Htop
↓
View System Resources
↓
Press F6 to Sort Processes
↓
Identify Resource-Intensive Processes
↓
Take Action (Q to Quit)

Glances - No Longer Recommended

Note: Glances was previously recommended but is no longer suggested due to the number of dependencies it requires. Use Htop instead for server resource monitoring.

Third-Party Monitoring Services

External monitoring services can be valuable for keeping track of server health and performance. These third-party services will normally install an agent on your server. This agent is used to gather data. However, you need to remember that these agents will also require resources.

Important Considerations:
  • Excessive resource usage by monitoring agents can impact the server's ability to serve your site visitors efficiently
  • It's crucial to strike a balance between monitoring needs and resource allocation for serving your visitors
  • Opting for monitoring solutions that offer lightweight agents or allow customization of monitoring parameters can help mitigate resource usage issues

Netdata Cloud

Netdata Cloud offers both free and paid server monitoring. Under pricing for home, you have the free or community edition of Netdata Cloud.

Disclaimer: No support is provided for third-party monitoring services. It is recommended that you initially install and test any third-party monitoring service on a development server, and not on your production server. Most third-party monitoring services have excellent documentation. The installation will normally involve adding a repository or running a script file on your server.

2. System Updates and Maintenance

Basic Update Commands

Regular system updates are essential for security and stability. Here are the basic commands for updating your Ubuntu server:

Update Package Lists

sudo apt update

Upgrade Installed Packages

sudo apt upgrade

Remove Unnecessary Packages

sudo apt autoremove

Combined Update Commands

You can chain these commands together for efficiency:

Interactive Combined Command

sudo apt update && sudo apt upgrade && sudo apt autoremove

Non-Interactive Combined Command (with -y flag)

sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y

System Update Process Flow

Update Package Lists
(apt update)
↓
Upgrade Packages
(apt upgrade)
↓
Remove Unused Dependencies
(apt autoremove)
↓
System Updated Successfully

Creating a Bash Alias for Updates

To simplify the update process, you can create a bash alias:

Edit Bash Aliases File

cd
nano .bash_aliases

Add the Following Alias

alias server_updates='sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y'

Activate the Alias

To activate the alias, either source the file or exit and log back in:

source ~/.bash_aliases

Or simply:

exit
Usage: After creating the alias, you can simply type server_updates to run all update commands automatically.

Creating an Update Script

For more control, you can create a dedicated update script:

Create the Script File

sudo nano server_updates.sh

Script Contents

#!/bin/bash # Check if script is run as root if [ "$EUID" -ne 0 ]; then echo "Please run this script with sudo or as root." exit fi # Update package lists echo "Updating package lists..." sudo apt update # Upgrade installed packages echo "Upgrading installed packages..." sudo apt upgrade -y # Remove unnecessary packages echo "Removing unnecessary packages..." sudo apt autoremove -y echo "Update, upgrade, and autoremove completed."

Make the Script Executable

sudo chmod +x server_updates.sh

Run the Script

sudo ./server_updates.sh
Note: The script includes the -y flag for non-interactive execution. You can remove -y if you prefer to confirm each action manually.

3. WordPress Updates and Permissions

Understanding WordPress Permissions

Proper file permissions are crucial for WordPress security and functionality. This section covers both default and hardened permission configurations.

Default PHP Pool User Permissions

These permissions allow WordPress to update plugins, themes, and core files directly from the dashboard:

Navigate to Site Directory

cd /var/www/example.com/

Check Current Permissions

sudo ls -l public_html/

Set Ownership

sudo chown -R PHP_POOL_USER:PHP_POOL_USER public_html/

Set Directory Permissions (770)

sudo find /var/www/example.com/public_html/ -type d -exec chmod 770 {} \;

Set File Permissions (660)

sudo find /var/www/example.com/public_html/ -type f -exec chmod 660 {} \;
Result: Open the WordPress Dashboard and all updates will run without issue.

Reload PHP-FPM

sudo systemctl reload php8.3-fpm

Hardened Permissions Configuration

For enhanced security, use hardened permissions that restrict write access except where necessary:

Set Ownership

sudo chown -R PHP_POOL_USER:PHP_POOL_USER public_html/

Set Hardened Directory Permissions (550)

sudo find /var/www/example.com/public_html/ -type d -exec chmod 550 {} \;

Set Hardened File Permissions (440)

sudo find /var/www/example.com/public_html/ -type f -exec chmod 440 {} \;

Allow Write Access to wp-content Directory (770)

sudo find /var/www/example.com/public_html/wp-content/ -type d -exec chmod 770 {} \;

Allow Write Access to wp-content Files (660)

sudo find /var/www/example.com/public_html/wp-content/ -type f -exec chmod 660 {} \;

WordPress Permission Workflow

Set Standard Permissions
(770/660)
↓
Run WordPress Updates
↓
Apply Hardened Permissions
(550/440 with wp-content 770/660)
↓
Reload PHP-FPM

Permission Update Workflow

When you need to run updates with hardened permissions in place:

Step 1: Change Permissions to Allow Updates

sudo find /var/www/example.com/public_html/ -type d -exec chmod 770 {} \;
sudo find /var/www/example.com/public_html/ -type f -exec chmod 660 {} \;

Step 2: Run Updates in WordPress Dashboard

Log into your WordPress dashboard and perform the necessary updates.

Step 3: Re-apply Hardened Permissions

sudo find /var/www/example.com/public_html/ -type d -exec chmod 550 {} \;
sudo find /var/www/example.com/public_html/ -type f -exec chmod 440 {} \;
sudo find /var/www/example.com/public_html/wp-content/ -type d -exec chmod 770 {} \;
sudo find /var/www/example.com/public_html/wp-content/ -type f -exec chmod 660 {} \;

Step 4: Reload PHP-FPM

sudo systemctl reload php8.3-fpm
Permission Type Directories Files Use Case
Standard 770 660 Full WordPress functionality, updates
Hardened 550 440 Read-only, enhanced security
wp-content (Hardened) 770 660 Allow uploads and dynamic content

4. Security Tools

ClamAV - Antivirus Scanner

ClamAV is an open-source antivirus engine designed for detecting malware and viruses on your server.

Installation

sudo apt update
sudo apt install clamav
Note: The ClamAV definition database will be updated automatically after installation.

Disable Automatic Updates (ClamAV-Freshclam)

If you prefer to update the database manually:

sudo systemctl stop clamav-freshclam
sudo systemctl disable clamav-freshclam

Manual Database Update

sudo freshclam

Running a Manual Scan

To scan a specific directory:

sudo clamscan -r /path/2/scan

Running ClamAV After Disabling Automatic Updates

Update the database first, then run the scan:

sudo freshclam
sudo clamscan -r /path/2/scan
Command Flags:
-r - Recursive scan (scan subdirectories)
/path/2/scan - Replace with your actual directory path

RKHunter - Rootkit Detection

RKHunter (Rootkit Hunter) is a security tool that scans for rootkits, backdoors, and possible local exploits.

Installation

sudo apt install rkhunter

Update RKHunter Database

sudo rkhunter --propupd

Run Complete System Check

sudo rkhunter --checkall --sk
Command Flags:
--checkall - Perform all checks
--sk, --skip-keypress - Skip keypress prompts for automation

View RKHunter Logs

To view the complete log file:

sudo cat /var/log/rkhunter.log

To view the log with pagination:

sudo less /var/log/rkhunter.log

Disable Automatic RKHunter Scans

If you prefer to run scans manually, remove the cron jobs:

Remove Daily Cron Job
cd /etc/cron.daily/
sudo rm rkhunter
Remove Weekly Cron Job
cd /etc/cron.weekly/
sudo rm rkhunter

Security Scanning Workflow

Install Security Tools
(ClamAV & RKHunter)
↓
Update Databases
(freshclam & rkhunter --propupd)
↓
Run Scans
(clamscan & rkhunter --checkall)
↓
Review Logs
(Check for threats)
↓
Take Action if Needed
Tool Purpose Update Command Scan Command
ClamAV Antivirus & Malware Detection sudo freshclam sudo clamscan -r /path
RKHunter Rootkit & Backdoor Detection sudo rkhunter --propupd sudo rkhunter --checkall --sk

5. Database Optimization

InnoDB Buffer Pool Monitoring

The InnoDB buffer pool is a crucial component for database performance. Monitoring its usage helps optimize memory allocation.

View InnoDB Buffer Pool Actual Memory Usage

SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_bytes_data';

Database Size Analysis

View All Database Sizes

SELECT table_schema AS "Database", ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS "Size (MB)" FROM information_schema.tables GROUP BY table_schema;

View InnoDB Database Sizes

SELECT table_schema AS "Database", SUM(data_length + index_length) / 1024 / 1024 AS "Size (MB)" FROM information_schema.tables WHERE engine = 'InnoDB' GROUP BY table_schema;

View Buffer Pool Usage by Database

SELECT table_schema AS "Database", SUM(data_length + index_length) / 1024 / 1024 AS "Size (MB)" FROM information_schema.tables WHERE engine = 'InnoDB' GROUP BY table_schema;

Table I/O Statistics

Understanding which tables are most active helps identify optimization opportunities.

View Database Table I/O Statistics

SELECT object_schema AS 'Database', object_name AS 'Table', COUNT_READ AS 'Reads', COUNT_WRITE AS 'Writes', SUM_TIMER_READ / 1000000000 AS 'Read Time (ms)', SUM_TIMER_WRITE / 1000000000 AS 'Write Time (ms)' FROM performance_schema.table_io_waits_summary_by_table ORDER BY SUM_TIMER_READ + SUM_TIMER_WRITE DESC;
Interpretation: This query shows which tables have the most read/write activity and which operations take the longest time. Tables at the top of the results are candidates for optimization.

MySQLTuner

MySQLTuner is a script that analyzes your MySQL/MariaDB installation and provides recommendations for optimization.

Running MySQLTuner

cd
cd MySQLTuner/
ls -l
sudo ./mysqltuner
Note: MySQLTuner needs to be run with sudo as it requires database login credentials to analyze the system.

Database Optimization Workflow

Monitor Buffer Pool Usage
↓
Analyze Database Sizes
↓
Review Table I/O Statistics
↓
Run MySQLTuner
↓
Implement Recommendations

6. Disk Space Management

Overview

Administering disk space on an Ubuntu server is an important task, as your server can crash if you run out of space. The process involves several tasks, including monitoring disk usage and then cleaning up unnecessary files.

Check Disk Space

View Disk Usage in Human-Readable Format

df -h
Flag Explanation:
-h - Display sizes in human-readable format (KB, MB, GB)

Cleanup Operations

Remove Unused Packages and Clean Package Cache

sudo apt autoremove && sudo apt clean
What This Does:
apt autoremove - Removes packages that were automatically installed as dependencies but are no longer needed
apt clean - Clears the local repository of downloaded package files

Cleanup System Logs

System logs can accumulate and consume significant disk space over time. You can limit journal logs to a specific time period:

sudo journalctl --vacuum-time=1days
Parameter: Replace "1days" with your preferred retention period (e.g., 2days, 1week, 2weeks)

Analyze Directory Sizes

List Directory Sizes Sorted by Size

cd /
du -ah --max-depth=1 | sort -h
Command Explanation:
du - Disk usage command
-a - Show all files and directories
-h - Human-readable format
--max-depth=1 - Only show first level subdirectories
sort -h - Sort by human-readable sizes

The largest directories are displayed at the bottom. You can change to any directory and run the du -ah command to further investigate space usage.

Disk Space Management Workflow

Check Disk Space
(df -h)
↓
Identify Large Directories
(du -ah | sort -h)
↓
Clean Packages
(apt autoremove && apt clean)
↓
Clean Logs
(journalctl --vacuum-time)
↓
Verify Space Recovered
Task Command Purpose
Check Disk Space df -h View overall disk usage
Clean Packages sudo apt autoremove && sudo apt clean Remove unused packages and cached files
Clean Logs sudo journalctl --vacuum-time=1days Remove old system logs
Analyze Directories du -ah --max-depth=1 | sort -h Find large directories

7. Performance Tuning

Nginx Backlog Configuration

The backlog parameter determines the maximum length of the queue for pending connections. A higher value can improve performance under high load.

Configuration Setting

backlog=2048

Implementation in Nginx Configuration

listen 80 backlog=2048; listen 443 ssl backlog=2048;
Explanation: This setting increases the connection queue size to 2048, allowing more simultaneous connection requests to be processed during traffic spikes.

PHP OPcache File Monitoring

Monitoring the number of PHP files per pool helps optimize OPcache settings for better performance.

Create OPcache Monitoring Script

nano opcache_files.sh

Script Contents

#!/bin/bash round_to_prime() { local value=$1 primes=(223 463 983 1979 3907 7963 16229 32531 65407 130987 262237 524524 1048793) closest=0 min_diff=$((value - primes[0])) for prime in "${primes[@]}"; do diff=$((value - prime)) if [ $diff -lt 0 ]; then closest=$prime break elif [ $diff -lt $min_diff ]; then min_diff=$diff closest=$prime fi done echo "$closest" } echo "" echo "Checking for opcache.max_accelerated_files directive in PHP pool files..." echo "" echo "The value used (actual value) will be the first number in the set of prime numbers" echo "223, 463, 983, 1979, 3907, 7963, 16229, 32531, 65407, 130987, 262237, 524521, 1048793" echo "that is greater than or equal to the configured value (value set)" echo "" POOL_DIR="/etc/php/8.3/fpm/pool.d/" for file in "$POOL_DIR"*.conf; do filename=$(basename "$file") if grep -q "opcache.max_accelerated_files" "$file"; then value=$(grep "opcache.max_accelerated_files" "$file" | awk -F'=' '{print $2}' | tr -d ' ') rounded_value=$(round_to_prime "$value") echo "File: $filename - opcache.max_accelerated_files: set value ($value) actual value ($rounded_value)" else echo "File: $filename - opcache.max_accelerated_files directive not found" fi done echo "" echo "Listing domain names and number of PHP files in public_html directories..." echo "" WWW_DIR="/var/www/" for domain_dir in "$WWW_DIR"*/ ; do if [ "$domain_dir" == "/var/www/html/" ]; then continue fi domain_name=$(basename "$domain_dir") public_html_dir="$domain_dir/public_html/" if [ -d "$public_html_dir" ]; then php_file_count=$(find "$public_html_dir" -type f -name "*.php" | wc -l) echo "WordPress site: $domain_name - PHP files: $php_file_count" else echo "Domain: $domain_name - public_html directory not found" fi done echo ""

Make Script Executable and Run

chmod +x opcache_files.sh
./opcache_files.sh
Script Purpose: This script analyzes each PHP-FPM pool configuration and counts the number of PHP files for each WordPress site. It also shows how OPcache rounds the max_accelerated_files value to the nearest prime number, which is important for optimal hash table performance.

Understanding OPcache Prime Numbers

PHP's OPcache uses prime numbers for the max_accelerated_files setting to optimize hash table distribution. The actual value used will be the first prime number that is greater than or equal to your configured value.

Configured Value Actual Prime Value Used Recommended For
200-223 223 Very small sites
400-463 463 Small sites
900-983 983 Small-medium sites
1900-1979 1979 Medium sites
3800-3907 3907 Medium-large sites
7800-7963 7963 Large sites
16000-16229 16229 Very large sites

Performance Optimization Workflow

Count PHP Files per Site
↓
Determine Appropriate OPcache Setting
↓
Configure Nginx Backlog
↓
Monitor Performance
↓
Adjust Settings as Needed

Best Practices Summary

Regular Maintenance Schedule

  • Daily: Monitor server resources with Htop, check external monitoring alerts
  • Weekly: Run system updates, check disk space usage
  • Monthly: Run security scans (ClamAV, RKHunter), review database performance
  • Quarterly: Review and optimize database settings, clean up old logs and backups

Security Best Practices

  • Keep all software updated with the latest security patches
  • Use hardened file permissions for WordPress installations when not performing updates
  • Regularly scan for malware and rootkits using ClamAV and RKHunter
  • Monitor server logs for suspicious activity
  • Implement external monitoring to detect downtime quickly

Performance Optimization Tips

  • Configure OPcache max_accelerated_files based on actual PHP file count
  • Adjust Nginx backlog settings for high-traffic sites
  • Optimize database buffer pool size based on database size and usage
  • Monitor and clean up disk space regularly to prevent server crashes
  • Use MySQLTuner recommendations to optimize database configuration
Important Reminder: Always test configuration changes on a development server before applying them to production. Keep regular backups of your server configuration and data.