VPS: Difference between revisions
Appearance
No edit summary |
No edit summary |
||
Line 3: | Line 3: | ||
''Self-hosted services and applications running on the ejfvps infrastructure'' | ''Self-hosted services and applications running on the ejfvps infrastructure'' | ||
== Applications == | == Applications == | ||
* '''Main Website''' - [https://ejfox.com ejfox.com] - Personal site and blog (Nuxt.js) | * '''Main Website''' - [https://ejfox.com ejfox.com] - Personal site and blog (Nuxt.js) | ||
* '''Archive Wiki''' - [https://archive.ejfox.com archive.ejfox.com] - MediaWiki knowledge base | * '''Archive Wiki''' - [https://archive.ejfox.com archive.ejfox.com] - MediaWiki knowledge base | ||
Line 10: | Line 10: | ||
* '''Personal APIs''' - ejfox.com/api/* - Self-quantification data endpoints | * '''Personal APIs''' - ejfox.com/api/* - Self-quantification data endpoints | ||
* '''[[ArchiveBox]]''' - [https://snap.ejfox.com snap.ejfox.com] - Web archiving and snapshot preservation | * '''[[ArchiveBox]]''' - [https://snap.ejfox.com snap.ejfox.com] - Web archiving and snapshot preservation | ||
* '''n8n''' - [https://n8n.tools.ejfox.com n8n.tools.ejfox.com] - Workflow automation and API orchestration | * '''n8n''' - [https://n8n.tools.ejfox.com n8n.tools.ejfox.com] - Workflow automation and API orchestration | ||
* '''Grafana''' - [https://grafana.tools.ejfox.com grafana.tools.ejfox.com] - System metrics and personal data dashboards | * '''Grafana''' - [https://grafana.tools.ejfox.com grafana.tools.ejfox.com] - System metrics and personal data dashboards | ||
Line 16: | Line 15: | ||
* '''Umami Analytics''' - [https://umami.tools.ejfox.com umami.tools.ejfox.com] - Privacy-focused web analytics | * '''Umami Analytics''' - [https://umami.tools.ejfox.com umami.tools.ejfox.com] - Privacy-focused web analytics | ||
== System Overview == | |||
== | === Hardware Specifications === | ||
{| class="wikitable" | |||
! Component !! Specification | |||
|- | |||
| '''Host System''' || Debian GNU/Linux 12 (bookworm) | |||
|- | |||
| '''Kernel''' || Linux 6.1.0-37-cloud-amd64 | |||
|- | |||
| '''Hostname''' || ejfvps | |||
|- | |||
| '''Architecture''' || x86_64 | |||
|- | |||
| '''CPU''' || Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2 cores) | |||
|- | |||
| '''Memory''' || 3.8GB RAM | |||
|- | |||
| '''Root Storage''' || 7.8GB (/dev/vda1) | |||
|- | |||
| '''Data Storage''' || 20GB (/dev/vdb1) | |||
|- | |||
| '''Data2 Storage''' || 49GB (/dev/vdc) | |||
|} | |||
=== Port Mapping === | === Port Mapping === | ||
Line 57: | Line 71: | ||
|} | |} | ||
== Docker Troubleshooting == | |||
=== Emergency Procedures === | |||
== | ==== Kill Runaway Docker Process ==== | ||
When dockerd consumes 100%+ CPU: | |||
<pre> | |||
< | |||
# Force kill Docker | # Force kill Docker | ||
sudo kill -9 $(pgrep dockerd) | sudo kill -9 $(pgrep dockerd) | ||
Line 200: | Line 83: | ||
sudo systemctl stop docker.socket | sudo systemctl stop docker.socket | ||
sudo systemctl stop containerd | sudo systemctl stop containerd | ||
</ | </pre> | ||
==== Disable Docker Bridge Networking ==== | |||
< | Prevents bridge networking conflicts on VPS: | ||
# | <pre> | ||
# Edit /etc/docker/daemon.json | |||
{ | { | ||
"iptables": false, | "iptables": false, | ||
Line 218: | Line 96: | ||
# Restart Docker | # Restart Docker | ||
sudo systemctl restart docker | sudo systemctl restart docker | ||
</ | </pre> | ||
==== Prevent Docker Auto-Restart ==== | |||
< | <pre> | ||
sudo systemctl stop docker docker.socket containerd | sudo systemctl stop docker docker.socket containerd | ||
sudo systemctl disable docker docker.socket containerd | sudo systemctl disable docker docker.socket containerd | ||
sudo systemctl mask docker # Prevents any activation | sudo systemctl mask docker # Prevents any activation | ||
</ | </pre> | ||
=== Disk Space Management === | === Disk Space Management === | ||
==== Quick | ==== Quick Cleanup Commands ==== | ||
< | <pre> | ||
# Nuclear Docker cleanup (removes ALL unused data) | # Nuclear Docker cleanup (removes ALL unused data) | ||
docker system prune -a --volumes --force | docker system prune -a --volumes --force | ||
Line 242: | Line 117: | ||
# Clean apt cache | # Clean apt cache | ||
sudo apt clean | sudo apt clean | ||
</pre> | |||
# Find | ==== Find Large Files Without du ==== | ||
When du hangs or is slow: | |||
<pre> | |||
# Find files over 100MB | |||
find /data2 -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | find /data2 -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | ||
==== Docker Overlay2 | # Sort directory by size | ||
ls -lhS /directory/ | |||
# Count Docker veth interfaces | |||
ip link show | grep veth | wc -l | |||
</pre> | |||
==== Docker Overlay2 Maintenance ==== | |||
Location: `/var/lib/docker/overlay2/` or `/data2/docker/overlay2/` | |||
<pre> | |||
< | |||
# Check Docker disk usage | # Check Docker disk usage | ||
docker system df | docker system df | ||
# Remove unused images | # Progressive cleanup | ||
docker | docker image prune -a # Remove unused images | ||
docker container prune # Remove stopped containers | |||
docker volume prune # Remove unused volumes | |||
docker builder prune # Remove build cache | |||
</pre> | |||
=== System Diagnostics === | |||
==== Process Monitoring ==== | |||
<pre> | |||
# System load | |||
top | |||
ps aux | grep -E "(docker|containerd)" | |||
# Memory status | |||
free -h | |||
# Disk status | |||
df -h | |||
</pre> | |||
==== Log Investigation ==== | |||
<pre> | |||
# Check time-specific logs | |||
sudo journalctl --since "2025-09-09 07:00" --until "2025-09-09 08:00" | |||
# Docker logs | |||
sudo journalctl -u docker -n 100 | |||
# | # System errors | ||
sudo journalctl -b -p err | |||
</pre> | |||
=== Recovery Methods === | |||
=== | ==== Console Keyboard Shortcuts ==== | ||
* '''Ctrl+Alt+F2 to F6''' - Switch TTY when console is flooded | |||
* '''Ctrl+C''' - Interrupt running process | |||
* '''Ctrl+Alt+Del''' - Force system reboot | |||
==== | ==== Volume Mount Recovery ==== | ||
When instance is inaccessible: | |||
# Stop broken instance | # Stop broken instance | ||
# Create minimal rescue instance | # Create minimal rescue instance | ||
# Detach root volume from broken instance | # Detach root volume from broken instance | ||
# Attach to rescue instance as | # Attach to rescue instance as secondary drive | ||
# Mount and | # Mount and repair: | ||
< | |||
<pre> | |||
sudo mount /dev/vdb1 /mnt | sudo mount /dev/vdb1 /mnt | ||
sudo chroot /mnt | sudo chroot /mnt | ||
# Fix | # Fix issues (disable services, edit configs, etc) | ||
systemctl disable problem-service | systemctl disable problem-service | ||
exit | exit | ||
sudo umount /mnt | sudo umount /mnt | ||
</ | </pre> | ||
=== Preventive Maintenance === | |||
=== | ==== Weekly Tasks ==== | ||
* Clean Docker: `docker system prune -a --volumes` | |||
* Check disk usage: `df -h` | |||
* Review Docker disk usage: `docker system df` | |||
==== Monthly Tasks ==== | |||
* Clean old logs: `sudo journalctl --vacuum-time=30d` | |||
* Update system packages: `sudo apt update && sudo apt upgrade` | |||
* Review container resource usage | |||
==== Critical Files ==== | |||
{| class="wikitable" | |||
! File/Directory !! Purpose | |||
|- | |||
| /etc/docker/daemon.json || Docker daemon configuration | |||
|- | |||
| /etc/systemd/system/docker.service.d/ || Docker service overrides | |||
|- | |||
| /var/lib/docker/ || Docker data directory | |||
|- | |||
| /etc/caddy/Caddyfile || Reverse proxy configuration | |||
|} | |||
=== Network Configuration === | |||
==== | ==== Docker Network Alternatives ==== | ||
When bridge networking fails: | |||
# | <pre> | ||
# Host networking (shares host network) | |||
docker run --network host image:tag | |||
# | # No network | ||
docker run --network none image:tag | |||
# | # Custom network | ||
docker network create mynet --driver bridge | |||
docker run --network mynet image:tag | |||
</pre> | |||
==== Firewall Management ==== | |||
With Docker iptables disabled: | |||
* Manage ports via Caddy reverse proxy | |||
* Use ufw for firewall rules | |||
* Manual iptables configuration if needed | |||
==== | === Monitoring Thresholds === | ||
{| class="wikitable" | |||
! Metric !! Warning Level !! Critical Level | |||
|- | |||
| Disk Usage || 85% || 95% | |||
|- | |||
| CPU Usage (sustained) || 70% || 90% | |||
|- | |||
| Memory Usage || 80% || 90% | |||
|- | |||
| Docker overlay2 size || 20GB || 30GB | |||
|} | |||
== Service-Specific Documentation == | |||
=== Core Services === | |||
=== | ==== Caddy (Reverse Proxy) ==== | ||
* '''Port:''' 80/443 | |||
* '''Config:''' /etc/caddy/Caddyfile | |||
* '''Features:''' Automatic SSL, HTTP/2, reverse proxy | |||
* '''Restart:''' `sudo systemctl restart caddy` | |||
==== | ==== Docker & Containerd ==== | ||
* '''Config:''' /etc/docker/daemon.json | |||
* '''Data:''' /data2/docker/ | |||
docker | * '''Status:''' `systemctl status docker` | ||
* '''Logs:''' `journalctl -u docker` | |||
==== PM2 Process Manager ==== | |||
* '''Status:''' `pm2 status` | |||
* '''Logs:''' `pm2 logs` | |||
* '''Save config:''' `pm2 save` | |||
* '''Startup script:''' `pm2 startup` | |||
=== Container Management === | |||
==== | ==== Essential Container Commands ==== | ||
<pre> | |||
# View all containers | |||
docker ps -a | |||
# Start specific service | |||
docker start container_name | |||
# View logs | |||
docker logs -f container_name | |||
# Restart with docker-compose | |||
cd /path/to/project | |||
docker-compose up -d | |||
docker | |||
# | # Rebuild container | ||
docker | docker-compose build --no-cache | ||
docker-compose up -d | |||
</pre> | |||
==== Backup Procedures ==== | |||
docker | * MediaWiki: Backup scripts in `/home/debian/mediawiki/` | ||
* Docker volumes: Located in `/data2/docker/volumes/` | |||
* Databases: Regular dumps to `/data/backups/` | |||
* Configuration: Version controlled in git repositories | |||
[[Category:System Administration]] | |||
Docker | [[Category:Docker]] | ||
[[Category:VPS Management]] | |||
Latest revision as of 13:43, 9 September 2025
VPS Documentation
Self-hosted services and applications running on the ejfvps infrastructure
Applications
- Main Website - ejfox.com - Personal site and blog (Nuxt.js)
- Archive Wiki - archive.ejfox.com - MediaWiki knowledge base
- Smallweb - *.ejfox.tools - Small self-hosted deno micro-apps
- Kuma Uptime Monitoring - status.tools.ejfox.com - Service availability tracking
- Personal APIs - ejfox.com/api/* - Self-quantification data endpoints
- ArchiveBox - snap.ejfox.com - Web archiving and snapshot preservation
- n8n - n8n.tools.ejfox.com - Workflow automation and API orchestration
- Grafana - grafana.tools.ejfox.com - System metrics and personal data dashboards
- Loki - loki.tools.ejfox.com - Log aggregation and analysis
- Umami Analytics - umami.tools.ejfox.com - Privacy-focused web analytics
System Overview
Hardware Specifications
Component | Specification |
---|---|
Host System | Debian GNU/Linux 12 (bookworm) |
Kernel | Linux 6.1.0-37-cloud-amd64 |
Hostname | ejfvps |
Architecture | x86_64 |
CPU | Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2 cores) |
Memory | 3.8GB RAM |
Root Storage | 7.8GB (/dev/vda1) |
Data Storage | 20GB (/dev/vdb1) |
Data2 Storage | 49GB (/dev/vdc) |
Port Mapping
Service | Internal Port | Public URL | Purpose |
---|---|---|---|
Main Website | 3013 | ejfox.com | Personal site & blog |
Personal APIs | 3013 | ejfox.com/api/* | Self-quantification endpoints |
MediaWiki | 8437 | archive.ejfox.com | Knowledge base |
ArchiveBox | 4040 | snap.ejfox.com | Web archiving |
Smallweb | 7777 | ejfox.tools | Development platform |
n8n | 5678 | n8n.tools.ejfox.com | Workflow automation |
Datasette | 8002 | datasette.tools.ejfox.com | Data exploration |
Capabilities | 9991 | capabilities.tools.ejfox.com | Skills tracking |
Grafana | 3333 | grafana.tools.ejfox.com | Metrics dashboards |
Loki | 3100 | loki.tools.ejfox.com | Log aggregation |
Umami | 3900 | umami.tools.ejfox.com | Web analytics |
Uptime Kuma | 54321 | status.tools.ejfox.com | Uptime monitoring |
Room302 | 3012 | room302.studio | Creative projects |
Docker Troubleshooting
Emergency Procedures
Kill Runaway Docker Process
When dockerd consumes 100%+ CPU:
# Force kill Docker sudo kill -9 $(pgrep dockerd) sudo systemctl stop docker sudo systemctl stop docker.socket sudo systemctl stop containerd
Disable Docker Bridge Networking
Prevents bridge networking conflicts on VPS:
# Edit /etc/docker/daemon.json { "iptables": false, "bridge": "none" } # Restart Docker sudo systemctl restart docker
Prevent Docker Auto-Restart
sudo systemctl stop docker docker.socket containerd sudo systemctl disable docker docker.socket containerd sudo systemctl mask docker # Prevents any activation
Disk Space Management
Quick Cleanup Commands
# Nuclear Docker cleanup (removes ALL unused data) docker system prune -a --volumes --force # Clean journal logs sudo journalctl --vacuum-time=3d # Clean apt cache sudo apt clean
Find Large Files Without du
When du hangs or is slow:
# Find files over 100MB find /data2 -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh # Sort directory by size ls -lhS /directory/ # Count Docker veth interfaces ip link show | grep veth | wc -l
Docker Overlay2 Maintenance
Location: `/var/lib/docker/overlay2/` or `/data2/docker/overlay2/`
# Check Docker disk usage docker system df # Progressive cleanup docker image prune -a # Remove unused images docker container prune # Remove stopped containers docker volume prune # Remove unused volumes docker builder prune # Remove build cache
System Diagnostics
Process Monitoring
# System load top ps aux | grep -E "(docker|containerd)" # Memory status free -h # Disk status df -h
Log Investigation
# Check time-specific logs sudo journalctl --since "2025-09-09 07:00" --until "2025-09-09 08:00" # Docker logs sudo journalctl -u docker -n 100 # System errors sudo journalctl -b -p err
Recovery Methods
Console Keyboard Shortcuts
- Ctrl+Alt+F2 to F6 - Switch TTY when console is flooded
- Ctrl+C - Interrupt running process
- Ctrl+Alt+Del - Force system reboot
Volume Mount Recovery
When instance is inaccessible:
- Stop broken instance
- Create minimal rescue instance
- Detach root volume from broken instance
- Attach to rescue instance as secondary drive
- Mount and repair:
sudo mount /dev/vdb1 /mnt sudo chroot /mnt # Fix issues (disable services, edit configs, etc) systemctl disable problem-service exit sudo umount /mnt
Preventive Maintenance
Weekly Tasks
- Clean Docker: `docker system prune -a --volumes`
- Check disk usage: `df -h`
- Review Docker disk usage: `docker system df`
Monthly Tasks
- Clean old logs: `sudo journalctl --vacuum-time=30d`
- Update system packages: `sudo apt update && sudo apt upgrade`
- Review container resource usage
Critical Files
File/Directory | Purpose |
---|---|
/etc/docker/daemon.json | Docker daemon configuration |
/etc/systemd/system/docker.service.d/ | Docker service overrides |
/var/lib/docker/ | Docker data directory |
/etc/caddy/Caddyfile | Reverse proxy configuration |
Network Configuration
Docker Network Alternatives
When bridge networking fails:
# Host networking (shares host network) docker run --network host image:tag # No network docker run --network none image:tag # Custom network docker network create mynet --driver bridge docker run --network mynet image:tag
Firewall Management
With Docker iptables disabled:
- Manage ports via Caddy reverse proxy
- Use ufw for firewall rules
- Manual iptables configuration if needed
Monitoring Thresholds
Metric | Warning Level | Critical Level |
---|---|---|
Disk Usage | 85% | 95% |
CPU Usage (sustained) | 70% | 90% |
Memory Usage | 80% | 90% |
Docker overlay2 size | 20GB | 30GB |
Service-Specific Documentation
Core Services
Caddy (Reverse Proxy)
- Port: 80/443
- Config: /etc/caddy/Caddyfile
- Features: Automatic SSL, HTTP/2, reverse proxy
- Restart: `sudo systemctl restart caddy`
Docker & Containerd
- Config: /etc/docker/daemon.json
- Data: /data2/docker/
- Status: `systemctl status docker`
- Logs: `journalctl -u docker`
PM2 Process Manager
- Status: `pm2 status`
- Logs: `pm2 logs`
- Save config: `pm2 save`
- Startup script: `pm2 startup`
Container Management
Essential Container Commands
# View all containers docker ps -a # Start specific service docker start container_name # View logs docker logs -f container_name # Restart with docker-compose cd /path/to/project docker-compose up -d # Rebuild container docker-compose build --no-cache docker-compose up -d
Backup Procedures
- MediaWiki: Backup scripts in `/home/debian/mediawiki/`
- Docker volumes: Located in `/data2/docker/volumes/`
- Databases: Regular dumps to `/data/backups/`
- Configuration: Version controlled in git repositories