System Logs: Finding and Analyzing What Your System is Telling You

System Logs: Finding and Analyzing What Your System is Telling You
min read

System Logs: Finding and Analyzing What Your System is Telling You

Your Linux system is constantly talking to you—documenting every startup, every error, every security event, and every system change. The problem? Most of us don't know how to listen. System logs are like a detailed diary of everything happening on your machine, and learning to read them is one of the most valuable skills you can develop as a Linux user.

I learned this the hard way when my server started randomly crashing at 3 AM. Without understanding logs, I was flying blind, restarting services and hoping for the best. Once I mastered log analysis, I could pinpoint the exact cause: a memory leak in a poorly configured service that only manifested under specific load conditions.

In this guide, I'll show you how to become fluent in your system's language, from navigating the /var/log directory to mastering journalctl for systemd systems. By the end, you'll be able to diagnose issues, track down problems, and understand your system's behavior like a seasoned administrator.

Understanding the Linux Logging Ecosystem

Linux logging isn't just one system—it's an ecosystem of different logging mechanisms working together:

Traditional Syslog System

- rsyslog/syslog-ng: The classic logging daemon

  • Log files: Plain text files in /var/log
  • Log rotation: Automatic cleanup and archiving

    • Modern systemd Journal

    - journald: Binary logging system

  • journalctl: Command-line tool for journal access
  • Persistent storage: Optional persistent logging

    • Application-Specific Logs

    - Web servers: Apache, Nginx logs

  • Databases: MySQL, PostgreSQL logs
  • Custom applications: Their own log formats

    • The `/var/log` Directory: Your System's Memory Bank

    Let's start with the traditional logging system. The /var/log directory is where most system and application logs live.

    Essential Log Files

    bash 
    # Explore the log directory structure
ls -la /var/log/

# Most important system logs
ls -la /var/log/ | grep -E "(syslog|messages|kern|auth|boot)"

    System-Wide Logs

    bash 
    # General system messages (Ubuntu/Debian)
sudo tail -f /var/log/syslog

# General system messages (CentOS/RHEL)
sudo tail -f /var/log/messages

# Kernel messages
sudo tail -f /var/log/kern.log

# Boot messages
sudo less /var/log/boot.log

    Security and Authentication Logs

    bash 
    # Authentication attempts (successful and failed)
sudo tail -f /var/log/auth.log        # Ubuntu/Debian
sudo tail -f /var/log/secure          # CentOS/RHEL

# Failed login attempts
sudo grep "Failed password" /var/log/auth.log

# Successful sudo usage
sudo grep "sudo:" /var/log/auth.log

    Application Logs

    bash 
    # Web server logs
sudo tail -f /var/log/apache2/access.log
sudo tail -f /var/log/apache2/error.log
sudo tail -f /var/log/nginx/access.log

# Package manager logs
sudo less /var/log/apt/history.log     # Ubuntu/Debian
sudo less /var/log/yum.log             # CentOS/RHEL

# Cron job logs
sudo tail -f /var/log/cron

    Understanding Log File Formats

    Most traditional log files follow a standard format:

    plaintext 
    Timestamp Hostname Service[PID]: Message

    Example:

    plaintext 
    Jan 15 10:30:45 myserver sshd[1234]: Accepted password for user from 192.168.1.100 port 22 ssh2

    Let's break this down:

  • Jan 15 10:30:45: Timestamp
  • myserver: Hostname
  • sshd[1234]: Service name and process ID
  • Message: What actually happened

    • Mastering journalctl: The Modern Way

    On systemd-based systems (most modern Linux distributions), journalctl is your primary tool for log analysis.

    Basic journalctl Usage

    bash 
    # View all journal entries
sudo journalctl

# View journal in real-time (like tail -f)
sudo journalctl -f

# Show only today's logs
sudo journalctl --since today

# Show logs from last hour
sudo journalctl --since "1 hour ago"

# Show logs for specific time range
sudo journalctl --since "2025-01-15 10:00:00" --until "2025-01-15 11:00:00"

    Filtering by Service and Unit

    bash 
    # View logs for specific service
sudo journalctl -u ssh
sudo journalctl -u apache2
sudo journalctl -u docker

# Follow logs for specific service
sudo journalctl -u nginx -f

# View logs for multiple services
sudo journalctl -u ssh -u apache2

    Filtering by Priority Level

    Journal entries have priority levels similar to syslog:

    bash 
    # Only show errors and critical messages
sudo journalctl -p err

# Show warnings and above (warning, error, critical)
sudo journalctl -p warning

# Priority levels:
# 0: Emergency (emerg)
# 1: Alert (alert)
# 2: Critical (crit)
# 3: Error (err)
# 4: Warning (warning)
# 5: Notice (notice)
# 6: Informational (info)
# 7: Debug (debug)

    Advanced journalctl Features

    bash 
    # Show reverse chronological order (newest first)
sudo journalctl -r

# Show only kernel messages
sudo journalctl -k

# Show boot messages for current boot
sudo journalctl -b

# Show boot messages for previous boot
sudo journalctl -b -1

# List all available boots
sudo journalctl --list-boots

# Show logs with specific field
sudo journalctl _PID=1234
sudo journalctl _COMM=sshd
sudo journalctl _UID=1000

# Output in JSON format
sudo journalctl -o json

# Show disk usage of journal
sudo journalctl --disk-usage

    Practical Log Analysis Scenarios

    Scenario 1: Investigating Failed Login Attempts

    bash 
    # Check for failed SSH attempts
sudo grep "Failed password" /var/log/auth.log

# Using journalctl (more detailed)
sudo journalctl -u ssh | grep "Failed password"

# Look for brute force patterns
sudo grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -nr

# Check for successful logins from suspicious IPs
sudo grep "Accepted password" /var/log/auth.log | tail -20

    Scenario 2: Diagnosing Service Startup Issues

    bash 
    # Check why a service failed to start
sudo systemctl status apache2
sudo journalctl -u apache2 --since "10 minutes ago"

# Look for configuration errors
sudo journalctl -u apache2 -p err

# Check dependencies
sudo journalctl -u apache2 --since today | grep -i "depend"

    Scenario 3: Investigating System Crashes

    bash 
    # Look for kernel panics or OOM (Out of Memory) kills
sudo journalctl -k | grep -i "panic\|oops\|kill"

# Check for hardware issues
sudo journalctl -k | grep -i "error\|fail"

# Look for memory issues
sudo journalctl | grep -i "out of memory\|oom"

# Check system reboot reasons
sudo journalctl | grep -i "reboot\|shutdown"

    Scenario 4: Web Server Troubleshooting

    bash 
    # Check Apache/Nginx error logs
sudo tail -f /var/log/apache2/error.log
sudo tail -f /var/log/nginx/error.log

# Analyze access patterns
sudo tail -100 /var/log/apache2/access.log | awk '{print $1}' | sort | uniq -c | sort -nr

# Find 404 errors
sudo grep " 404 " /var/log/apache2/access.log

# Monitor real-time access
sudo tail -f /var/log/apache2/access.log | grep -v "\.css\|\.js\|\.ico"

    Log Rotation and Management

    Log files can grow very large over time. Linux uses log rotation to manage this:

    Understanding logrotate

    bash 
    # View logrotate configuration
sudo cat /etc/logrotate.conf

# View specific service rotation config
sudo ls /etc/logrotate.d/
sudo cat /etc/logrotate.d/apache2

# Manually trigger log rotation
sudo logrotate -f /etc/logrotate.conf

# Test logrotate configuration
sudo logrotate -d /etc/logrotate.conf

    Creating Custom Log Rotation

    bash 
    # Create custom logrotate config
sudo nano /etc/logrotate.d/myapp

# Example configuration:
/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    create 0644 www-data www-data
    postrotate
        systemctl reload myapp
    endscript
}

    Journal Size Management

    bash 
    # Check journal disk usage
sudo journalctl --disk-usage

# Vacuum old journal entries
sudo journalctl --vacuum-time=7d    # Keep only 7 days
sudo journalctl --vacuum-size=500M  # Keep only 500MB

# Configure persistent journal size
sudo nano /etc/systemd/journald.conf
# Add: SystemMaxUse=500M

    Advanced Log Analysis Techniques

    Using grep and awk for Log Analysis

    bash 
    # Find all errors in the last 24 hours
sudo journalctl --since "24 hours ago" | grep -i error

# Extract IP addresses from auth log
sudo grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq

# Count connections per IP
sudo awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -10

# Find large requests
sudo awk '{if ($10 > 1000000) print $0}' /var/log/apache2/access.log

    Creating Log Analysis Scripts

    bash 
    #!/bin/bash
# System Health Check Script

echo "=== System Log Analysis Report ==="
echo "Generated on: $(date)"
echo

echo "=== Recent Critical Errors ==="
sudo journalctl -p crit --since "24 hours ago" --no-pager | tail -10

echo -e "\n=== Failed Login Attempts ==="
sudo grep "Failed password" /var/log/auth.log | tail -5

echo -e "\n=== Service Failures ==="
sudo journalctl -p err --since "24 hours ago" | grep -i "failed\|error" | tail -5

echo -e "\n=== Disk Space Warnings ==="
sudo journalctl | grep -i "no space\|disk full" | tail -5

echo -e "\n=== Memory Issues ==="
sudo journalctl | grep -i "out of memory\|oom killer" | tail -5

    Real-Time Log Monitoring

    bash 
    # Monitor multiple logs simultaneously
sudo tail -f /var/log/syslog /var/log/auth.log

# Use multitail for better visualization (install first)
sudo apt install multitail
sudo multitail /var/log/syslog /var/log/auth.log

# Monitor journal with filtering
sudo journalctl -f | grep -E "(error|fail|critical)"

    Security Log Analysis

    Detecting Intrusion Attempts

    bash 
    # Look for suspicious login patterns
sudo grep "authentication failure" /var/log/auth.log

# Check for privilege escalation attempts
sudo grep "sudo:" /var/log/auth.log | grep "incorrect password"

# Look for unusual user account activities
sudo journalctl _COMM=login | grep -v "$(whoami)"

# Check for port scanning attempts (if logged)
sudo journalctl | grep -i "port scan\|connection refused"

    Monitoring File System Changes

    bash 
    # Check for package installations/removals
sudo grep -E "(install|remove|upgrade)" /var/log/apt/history.log | tail -10

# Look for configuration changes
sudo journalctl | grep -i "config" | tail -10

# Monitor cron job execution
sudo grep "CRON" /var/log/syslog | tail -10

    Performance Log Analysis

    Identifying Performance Issues

    bash 
    # Look for slow processes
sudo journalctl | grep -i "slow\|timeout"

# Check for memory pressure
sudo journalctl -k | grep -i "memory\|swap"

# Look for I/O issues
sudo journalctl -k | grep -i "i/o\|disk"

# Check for network issues
sudo journalctl | grep -i "network\|connection"

    Database Log Analysis

    bash 
    # MySQL slow query log
sudo tail -f /var/log/mysql/mysql-slow.log

# PostgreSQL logs
sudo tail -f /var/log/postgresql/postgresql-*.log

# Look for connection issues
sudo grep -i "connection" /var/log/mysql/error.log

    Centralized Logging Setup

    For multiple servers, centralized logging is essential:

    Setting Up Remote Logging

    bash 
    # Configure rsyslog to send logs to remote server
sudo nano /etc/rsyslog.conf

# Add this line to send logs to remote server:
# *.* @@logserver.example.com:514

# Restart rsyslog
sudo systemctl restart rsyslog

    Journal Forwarding

    bash 
    # Configure journal to forward to syslog
sudo nano /etc/systemd/journald.conf

# Add:
# ForwardToSyslog=yes
# ForwardToWall=no

# Restart journald
sudo systemctl restart systemd-journald

    Troubleshooting Common Log Issues

    Log Files Not Being Created

    bash 
    # Check if logging service is running
sudo systemctl status rsyslog
sudo systemctl status systemd-journald

# Check permissions
ls -la /var/log/

# Check disk space
df -h /var/log/

# Check SELinux context (if applicable)
ls -Z /var/log/

    Journal Not Persistent

    bash 
    # Create persistent journal directory
sudo mkdir -p /var/log/journal

# Set proper permissions
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Restart journald
sudo systemctl restart systemd-journald

    Large Log Files

    bash 
    # Find large log files
sudo find /var/log -type f -size +100M -exec ls -lh {} \;

# Compress old logs manually
sudo gzip /var/log/old-logfile.log

# Clean up journal
sudo journalctl --vacuum-size=100M

    Log Analysis Tools and Utilities

    Command-Line Tools

    bash 
    # Install useful log analysis tools
sudo apt install ccze lnav multitail

# Colorize log output
sudo tail -f /var/log/syslog | ccze -A

# Interactive log navigator
sudo lnav /var/log/syslog

# Better tail for multiple files
sudo multitail /var/log/syslog /var/log/auth.log

    Creating Custom Alerts

    bash 
    #!/bin/bash
# Simple log alert script

LOGFILE="/var/log/auth.log"
ALERT_EMAIL="admin@example.com"

# Check for failed login attempts in last 5 minutes
FAILED_LOGINS=$(sudo grep "Failed password" $LOGFILE | grep "$(date '+%b %d %H:%M' -d '5 minutes ago')" | wc -l)

if [ $FAILED_LOGINS -gt 10 ]; then
    echo "ALERT: $FAILED_LOGINS failed login attempts in last 5 minutes" | mail -s "Security Alert" $ALERT_EMAIL
fi

    Performance Considerations

    Log Storage Optimization

    bash 
    # Check log directory sizes
sudo du -sh /var/log/*

# Implement log compression
sudo nano /etc/logrotate.d/myservice
# Add: compress

# Use journal vacuum regularly
sudo journalctl --vacuum-time=30d

# Consider remote log storage for large systems

    Log Parsing Performance

    bash 
    # Use appropriate tools for large files
# grep is faster for simple patterns
# awk is better for field extraction
# sed is good for replacements

# For very large files, consider:
sudo zcat /var/log/old-log.gz | grep "pattern"

    Best Practices for Log Management

    1. Regular Log Review

    bash 
    # Create a weekly log review script
#!/bin/bash
echo "Weekly Log Summary for $(hostname)"
echo "=================================="

echo "Critical Errors:"
sudo journalctl -p crit --since "1 week ago" | wc -l

echo "Failed Logins:"
sudo grep "Failed password" /var/log/auth.log | wc -l

echo "Service Restarts:"
sudo journalctl | grep "Started\|Stopped" | wc -l

    2. Automated Log Analysis

    bash 
    # Set up daily log analysis cron job
sudo crontab -e

# Add:
# 0 6 * * * /usr/local/bin/daily-log-check.sh

    3. Security Considerations

    - Protect log files with appropriate permissions

  • Monitor log file integrity
  • Implement log forwarding for critical systems
  • Regular backup of important logs

    • Quick Reference: Essential Log Commands

    bash 
    # Traditional Logs
sudo tail -f /var/log/syslog           # Follow system log
sudo grep -i error /var/log/syslog    # Find errors
sudo less /var/log/auth.log           # View auth log

# Journal Commands
sudo journalctl -f                     # Follow journal
sudo journalctl -u service-name       # Service logs
sudo journalctl --since "1 hour ago"  # Time-based filter
sudo journalctl -p err                # Error priority
sudo journalctl -k                    # Kernel messages

# Log Analysis
sudo grep "pattern" /var/log/*        # Search all logs
sudo journalctl | grep -i "error"     # Find errors in journal
sudo awk '{print $1}' logfile | sort | uniq -c    # Count occurrences

# Log Management
sudo logrotate -f /etc/logrotate.conf      # Force rotation
sudo journalctl --vacuum-size=100M        # Clean journal
sudo journalctl --disk-usage              # Check journal size

    What's Next?

    Understanding system logs is crucial for system administration, but it's just part of the bigger picture. Next, we'll explore network configuration and troubleshooting—another essential skill that often requires log analysis to diagnose connectivity issues, security problems, and performance bottlenecks.

    The combination of process management, log analysis, and network troubleshooting forms the foundation of Linux system administration. Master these three areas, and you'll be able to diagnose and resolve most system issues.

    Key Takeaways

    - Know your log locations: /var/log for traditional logs, journalctl for systemd

  • Use appropriate tools: grep, awk, and journalctl filters for different scenarios
  • Monitor proactively: Don't wait for problems to find you
  • Understand log rotation: Prevent disk space issues with proper log management
  • Practice regularly: Log analysis skills improve with experience
  • Security awareness: Logs are your first line of defense against intrusions

    Remember: Your system is constantly telling you what's happening—learning to listen is what separates good administrators from great ones. The logs don't lie, and they often contain the exact answer to your problem if you know how to read them.

    ---

    • 🚀 Continue Your Linux Journey

    This is Part 12 of our comprehensive Linux mastery series - entering Advanced Skills!

    Previous: Automation with Cron - Master task scheduling and automation

    Next: Network Configuration - Learn networking and connectivity troubleshooting

    📚 Complete Linux Series Navigation

    Advanced Skills Begin!

  • Part 12: System Logs Analysis ← You are here
  • Part 13: Network Configuration
  • Part 14: Systemd Deep Dive
  • Part 15: SSH Security
  • Part 16: Filesystem Hierarchy
  • Part 17: Firewalls & Security
  • Part 18: Storage Management
  • Part 19: Performance Monitoring

    Ready for Network Troubleshooting? Continue with network configuration to master connectivity diagnosis!

    • 🔗 Related Troubleshooting Topics

    - Process Management

  • Network Configuration
  • Performance Monitoring

    ---

    Ready to troubleshoot network connectivity issues? Next, we'll dive into Linux network configuration and troubleshooting, where log analysis skills will prove invaluable for diagnosing connectivity problems.

    • • Made With Love on