Jump to content

VPS: Difference between revisions

From Archive
← Older editNewer edit →
No edit summary
No edit summary
Line 182: Line 182:
* '''Stopped processes:''' Discord bot, capabilities service
* '''Stopped processes:''' Discord bot, capabilities service
* '''Log management:''' Automated log rotation
* '''Log management:''' Automated log rotation
---
== Docker Troubleshooting Guide ==
=== Common Docker Issues and Solutions ===
==== High CPU Usage by Docker ====
'''Symptoms:''' dockerd process consuming 100%+ CPU, system unresponsive, SSH slow or failing
'''Quick Fix:'''
<syntaxhighlight lang="bash">
# Force kill Docker
sudo kill -9 $(pgrep dockerd)
sudo systemctl stop docker
sudo systemctl stop docker.socket
sudo systemctl stop containerd
</syntaxhighlight>
==== Docker Bridge Networking Issues ====
'''Problem:''' Default bridge network conflicts with VPS network configuration
'''Solution:''' Disable Docker's bridge networking
<syntaxhighlight lang="bash">
# Create/edit Docker daemon config
sudo nano /etc/docker/daemon.json
# Add this configuration:
{
  "iptables": false,
  "bridge": "none"
}
# Restart Docker
sudo systemctl restart docker
</syntaxhighlight>
==== Docker Won't Stay Dead ====
'''Problem:''' Docker restarts automatically even after stopping
'''Solution:''' Disable all Docker services and sockets
<syntaxhighlight lang="bash">
sudo systemctl stop docker docker.socket containerd
sudo systemctl disable docker docker.socket containerd
sudo systemctl mask docker  # Prevents any activation
</syntaxhighlight>
=== Disk Space Management ===
==== Quick Space Recovery ====
<syntaxhighlight lang="bash">
# Nuclear Docker cleanup (removes ALL unused data)
docker system prune -a --volumes --force
# Clean journal logs
sudo journalctl --vacuum-time=3d
# Clean apt cache
sudo apt clean
# Find large files (when du doesn't work)
find /data2 -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh
ls -lhS /directory/  # Sort by size without du
</syntaxhighlight>
==== Docker Overlay2 Cleanup ====
Docker's overlay2 directory accumulates layers over time. Location: `/var/lib/docker/overlay2/` or `/data2/docker/overlay2/`
Regular maintenance:
<syntaxhighlight lang="bash">
# Check Docker disk usage
docker system df
# Remove unused images
docker image prune -a
# Remove stopped containers
docker container prune
# Remove unused volumes
docker volume prune
</syntaxhighlight>
=== Emergency Recovery Procedures ===
==== When Console is Flooded with Errors ====
'''Try these keyboard combinations:'''
* `Ctrl+Alt+F2` through `F6` - Switch to different TTY
* `Ctrl+C` repeatedly - Interrupt running process
* `Ctrl+Alt+Del` - Force system reboot
==== Creating Recovery Access ====
'''Option 1: Volume Mount Method'''
# Stop broken instance
# Create minimal rescue instance
# Detach root volume from broken instance
# Attach to rescue instance as /dev/vdb
# Mount and fix:
<syntaxhighlight lang="bash">
sudo mount /dev/vdb1 /mnt
sudo chroot /mnt
# Fix whatever is broken
systemctl disable problem-service
exit
sudo umount /mnt
</syntaxhighlight>
'''Option 2: Rebuild Instance'''
* Preserves attached data volumes
* Fresh OS installation
* Keeps IP address
* Requires reinstalling services
=== System Diagnostics ===
==== Essential Diagnostic Commands ====
<syntaxhighlight lang="bash">
# System load and processes
top
htop
ps aux | grep -E "(docker|high-cpu-process)"
# Disk I/O issues
iostat -x 1 3
iotop
# Network interfaces
ip link show | grep veth | wc -l  # Count Docker interfaces
# Memory status
free -h
# Disk usage (alternatives to du)
df -h
ls -lhS /directory/
find /path -type f -size +100M -exec ls -lh {} \;
</syntaxhighlight>
==== Log Investigation ====
<syntaxhighlight lang="bash">
# Check system logs for time period
sudo journalctl --since "2025-09-09 07:00" --until "2025-09-09 08:00"
# Docker-specific logs
sudo journalctl -u docker -n 100
# Check for errors
sudo journalctl -b -p err
# When logs show nothing (system too broken to log)
# This indicates severe resource exhaustion
</syntaxhighlight>
=== Preventive Measures ===
==== Regular Maintenance Tasks ====
<syntaxhighlight lang="bash">
# Weekly: Clean Docker
docker system prune -a --volumes
# Monthly: Clean logs
sudo journalctl --vacuum-time=30d
# Monitor disk usage
df -h
docker system df
</syntaxhighlight>
==== Critical Configuration Files ====
* `/etc/docker/daemon.json` - Docker daemon configuration
* `/etc/systemd/system/docker.service.d/` - Docker service overrides
* `/var/lib/docker/` - Docker data directory (can be symlinked to larger disk)
==== Resource Monitoring ====
Set up alerts for:
* Disk usage above 90%
* CPU usage sustained above 80%
* Memory usage above 85%
* Docker overlay2 directory size
=== Network Configuration for VPS ===
==== Alternative Docker Network Modes ====
When bridge networking causes issues:
<syntaxhighlight lang="bash">
# Host networking (shares host network stack)
docker run --network host image:tag
# None (no network)
docker run --network none image:tag
# Custom networks (create your own)
docker network create mynet --driver bridge
docker run --network mynet image:tag
</syntaxhighlight>
==== Firewall Considerations ====
Docker manipulates iptables by default. To prevent this:
* Set `"iptables": false` in daemon.json
* Manage firewall rules manually with ufw or iptables
* Use reverse proxy (Caddy/Nginx) for port management

Revision as of 13:40, 9 September 2025

VPS Documentation

Self-hosted services and applications running on the ejfvps infrastructure

Applications


Architecture

Domain Strategy

  • ejfox.com - Main personal brand and API endpoints
  • ejfox.tools - Development tools and utilities subdomain
  • *.tools.ejfox.com - Individual tool subdomains for organized access, powered by smallweb
  • archive.ejfox.com - Knowledge management and documentation
  • room302.studio - Studio work

Port Mapping

Service Internal Port Public URL Purpose
Main Website 3013 ejfox.com Personal site & blog
Personal APIs 3013 ejfox.com/api/* Self-quantification endpoints
MediaWiki 8437 archive.ejfox.com Knowledge base
ArchiveBox 4040 snap.ejfox.com Web archiving
Smallweb 7777 ejfox.tools Development platform
n8n 5678 n8n.tools.ejfox.com Workflow automation
Datasette 8002 datasette.tools.ejfox.com Data exploration
Capabilities 9991 capabilities.tools.ejfox.com Skills tracking
Grafana 3333 grafana.tools.ejfox.com Metrics dashboards
Loki 3100 loki.tools.ejfox.com Log aggregation
Umami 3900 umami.tools.ejfox.com Web analytics
Uptime Kuma 54321 status.tools.ejfox.com Uptime monitoring
Room302 3012 room302.studio Creative projects


System Overview

Host System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-37-cloud-amd64
Hostname: ejfvps
Architecture: x86_64
CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2 cores)
Memory: 3.8GB RAM
Storage:

  • Root: 7.8GB (/dev/vda1)
  • Data: 20GB (/dev/vdb1)
  • Data2: 49GB (/dev/vdc)

Core Infrastructure Services

Reverse Proxy & SSL

  • Caddy - Modern web server with automatic HTTPS
    • Port: 80/443
    • Config: /etc/caddy/Caddyfile
    • Features: Automatic SSL certificates, HTTP/2, reverse proxy

Container Orchestration

  • Docker Engine - Container runtime
  • Docker Compose - Multi-container application definitions
  • Containerd - Low-level container runtime

System Monitoring

  • Cloudflared - Cloudflare tunnel service
  • Fail2ban - Intrusion prevention system
  • Monit - System monitoring daemon

Application Stack

Analytics & Monitoring

Umami Analytics

Primary Instance:

  • Image: ghcr.io/umami-software/umami:postgresql-latest

Grafana Monitoring Stack

  • Grafana - Visualization dashboard
    • Image: grafana/grafana:latest
    • Plugins: ClickHouse, GitHub, Loki Explorer, Pyroscope, JSON datasource
  • Loki - Log aggregation system
    • Image: grafana/loki:latest

Uptime Monitoring

  • Uptime Kuma - Self-hosted monitoring tool

Content Management

MediaWiki

  • Image: mediawiki:latest
  • Port: 8437 → 80
  • Database: SQLite (multiple databases)
  • Extensions: AWS S3 integration
  • Skin: Citizen theme
  • Volumes:
    • Data: /home/debian/mediawiki/data
    • Config: /home/debian/mediawiki/LocalSettings.php

ArchiveBox

  • Image: archivebox/archivebox:latest
  • Features: Web archiving, full-text search via Sonic
  • Components:
    • Main archiver
    • Background scheduler (ofelia-based)
    • Sonic search backend
    • noVNC remote browser control

Development & Automation

Smallweb

  • Image: ghcr.io/pomdtr/smallweb:latest
  • Ports: 7777, 2222
  • Purpose: Lightweight web app hosting platform
  • Projects: Multiple TypeScript/Deno applications including:
    • Health dashboard and webhooks
    • GitHub integrations
    • Location tracking
    • Screenshot generation
    • Weather apps
    • Mastodon integration

n8n Workflow Automation

  • Image: docker.n8n.io/n8nio/n8n

Retroscope

  • Custom Application - retroscope
  • Scheduler: mcuadros/ofelia:latest
  • Status: Healthy, with background job scheduling

Personal Websites

ejfox.com

  • Custom Build - ejfoxcom_ejfox
  • Port: 3013 → 3013
  • Size: 1.84GB image
  • Status: Production website

room302.studio

  • Custom Build - website_room302
  • Port: 3012 → 3012
  • Size: 1.82GB image
  • Status: Secondary website

Network Configuration

Backup Strategy

  • MediaWiki has backup/restore scripts
  • Database files stored in persistent volumes
  • Configuration files version controlled

Process Management

PM2

  • PM2 modules: Auto-pull, log rotation
  • Stopped processes: Discord bot, capabilities service
  • Log management: Automated log rotation


---

Docker Troubleshooting Guide

Common Docker Issues and Solutions

High CPU Usage by Docker

Symptoms: dockerd process consuming 100%+ CPU, system unresponsive, SSH slow or failing

Quick Fix: <syntaxhighlight lang="bash">

  1. Force kill Docker

sudo kill -9 $(pgrep dockerd) sudo systemctl stop docker sudo systemctl stop docker.socket sudo systemctl stop containerd </syntaxhighlight>

Docker Bridge Networking Issues

Problem: Default bridge network conflicts with VPS network configuration

Solution: Disable Docker's bridge networking <syntaxhighlight lang="bash">

  1. Create/edit Docker daemon config

sudo nano /etc/docker/daemon.json

  1. Add this configuration:

{ 

 "iptables": false,
 "bridge": "none"

}

  1. Restart Docker

sudo systemctl restart docker </syntaxhighlight>

Docker Won't Stay Dead

Problem: Docker restarts automatically even after stopping

Solution: Disable all Docker services and sockets <syntaxhighlight lang="bash"> sudo systemctl stop docker docker.socket containerd sudo systemctl disable docker docker.socket containerd sudo systemctl mask docker # Prevents any activation </syntaxhighlight>

Disk Space Management

Quick Space Recovery

<syntaxhighlight lang="bash">

  1. Nuclear Docker cleanup (removes ALL unused data)

docker system prune -a --volumes --force

  1. Clean journal logs

sudo journalctl --vacuum-time=3d

  1. Clean apt cache

sudo apt clean

  1. Find large files (when du doesn't work)

find /data2 -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh ls -lhS /directory/ # Sort by size without du </syntaxhighlight>

Docker Overlay2 Cleanup

Docker's overlay2 directory accumulates layers over time. Location: `/var/lib/docker/overlay2/` or `/data2/docker/overlay2/`

Regular maintenance: <syntaxhighlight lang="bash">

  1. Check Docker disk usage

docker system df

  1. Remove unused images

docker image prune -a

  1. Remove stopped containers

docker container prune

  1. Remove unused volumes

docker volume prune </syntaxhighlight>

Emergency Recovery Procedures

When Console is Flooded with Errors

Try these keyboard combinations:

  • `Ctrl+Alt+F2` through `F6` - Switch to different TTY
  • `Ctrl+C` repeatedly - Interrupt running process
  • `Ctrl+Alt+Del` - Force system reboot

Creating Recovery Access

Option 1: Volume Mount Method

  1. Stop broken instance
  2. Create minimal rescue instance
  3. Detach root volume from broken instance
  4. Attach to rescue instance as /dev/vdb
  5. Mount and fix:

<syntaxhighlight lang="bash"> sudo mount /dev/vdb1 /mnt sudo chroot /mnt

  1. Fix whatever is broken

systemctl disable problem-service exit sudo umount /mnt </syntaxhighlight>

Option 2: Rebuild Instance

  • Preserves attached data volumes
  • Fresh OS installation
  • Keeps IP address
  • Requires reinstalling services

System Diagnostics

Essential Diagnostic Commands

<syntaxhighlight lang="bash">

  1. System load and processes

top htop ps aux | grep -E "(docker|high-cpu-process)"

  1. Disk I/O issues

iostat -x 1 3 iotop

  1. Network interfaces

ip link show | grep veth | wc -l # Count Docker interfaces

  1. Memory status

free -h

  1. Disk usage (alternatives to du)

df -h ls -lhS /directory/ find /path -type f -size +100M -exec ls -lh {} \; </syntaxhighlight>

Log Investigation

<syntaxhighlight lang="bash">

  1. Check system logs for time period

sudo journalctl --since "2025-09-09 07:00" --until "2025-09-09 08:00"

  1. Docker-specific logs

sudo journalctl -u docker -n 100

  1. Check for errors

sudo journalctl -b -p err

  1. When logs show nothing (system too broken to log)
  2. This indicates severe resource exhaustion

</syntaxhighlight>

Preventive Measures

Regular Maintenance Tasks

<syntaxhighlight lang="bash">

  1. Weekly: Clean Docker

docker system prune -a --volumes

  1. Monthly: Clean logs

sudo journalctl --vacuum-time=30d

  1. Monitor disk usage

df -h docker system df </syntaxhighlight>

Critical Configuration Files

  • `/etc/docker/daemon.json` - Docker daemon configuration
  • `/etc/systemd/system/docker.service.d/` - Docker service overrides
  • `/var/lib/docker/` - Docker data directory (can be symlinked to larger disk)

Resource Monitoring

Set up alerts for:

  • Disk usage above 90%
  • CPU usage sustained above 80%
  • Memory usage above 85%
  • Docker overlay2 directory size

Network Configuration for VPS

Alternative Docker Network Modes

When bridge networking causes issues: <syntaxhighlight lang="bash">

  1. Host networking (shares host network stack)

docker run --network host image:tag

  1. None (no network)

docker run --network none image:tag

  1. Custom networks (create your own)

docker network create mynet --driver bridge docker run --network mynet image:tag </syntaxhighlight>

Firewall Considerations

Docker manipulates iptables by default. To prevent this:

  • Set `"iptables": false` in daemon.json
  • Manage firewall rules manually with ufw or iptables
  • Use reverse proxy (Caddy/Nginx) for port management
Retrieved from "https://archive.ejfox.com/index.php?title=VPS&oldid=181"