Server Monitoring & Observability

Context & Overview

In high-concurrency game hosting and virtual server environments, operational incidents often manifest abruptly — a runaway Java thread or saturated disk I/O can degrade latency for hundreds of active players within seconds. Heavy commercial monitoring agents introduced undesirable memory footprints on budget VPS slices. This necessitated a bespoke, lightweight telemetry collector and real-time dispatcher engineered specifically for host health and rapid discord-centric alert delivery.

My Role & Responsibilities

• Designed the architecture for distributed telemetry polling across multiple host nodes.
• Developed low-overhead shell collectors querying /proc and kernel counters directly to prevent agent CPU hogging.
• Built a centralized aggregation service using Node.js and Redis pub/sub.
• Created intelligent webhook alert pipelines with debounce logic to prevent notification storms during transient spikes.
• Configured automated systemd service isolation and auto-restart policies.

The Problem & Operational Challenges

• Resource Overhead: Heavy monitoring suites (Prometheus/Grafana node_exporter stacks) consumed up to 300MB RAM per host, which was impractical for lightweight compute nodes.
• Alert Fatigue: Raw threshold alerting caused rapid alert spam whenever memory spiked briefly during world saves or startup sequences.
• Delayed Response Times: Operational engineers were frequently notified of outages by customer complaints rather than proactive telemetry alarms.

Architectural Approach & Engineering Decisions

1. Lightweight Shell & Proc Collectors: Replaced heavy metric agents with lean POSIX shell scripts that parse /proc/stat, /proc/meminfo, and disk queues, outputting compact JSON payloads every 2 seconds.
2. Redis In-Memory Buffering & Debouncing: Centralized metric ingest writes directly to Redis timeseries rings. Alert evaluations require a metric to exceed safety thresholds across 3 consecutive cycles before dispatching an alarm, filtering out 98% of transient spikes.
3. Structured Webhook Integration: Formatted rich embeds directed to private Discord operations channels, featuring color-coded severity states (Warning, Critical, Resolved), affected host tags, and recommended triage commands.

System Architecture

[ Host Node 01 ]                [ Host Node 02 ]
  │ Direct /proc parsing          │ Direct /proc parsing
  ▼ (Every 2s via Systemd timer)  ▼
[ Local Agent Daemon ]          [ Local Agent Daemon ]
        │                               │
        └──────────────┬────────────────┘
                       ▼ (Secure WebSocket / HTTPS)
              [ Central Ingest API ]
                       │
                       ▼
              [ Redis Cache / Buffer ]
                       │
             [ Alert Evaluation Engine ]
             (Threshold & Debounce Checks)
                       │
        ┌──────────────┴──────────────┐
        ▼                             ▼
 [ Operations Dashboard ]   [ Discord Webhook Pipeline ]
 (Real-time WS Broadcast)    (Actionable Alert Dispatch)

Public Status Page

The same collector feeds a public status page at status.arqonara.com. It turns internal telemetry into something a visitor can read without asking anyone:

• Two independent sources. Host state arrives from the agent heartbeat running on each node, while customer-facing endpoints are checked separately with HTTP and TCP probes from the status host. A node that stops reporting and a port that stops answering are two different failures, and the page distinguishes them.
• History, not just a light. Each service keeps an uptime history and an incident feed, so a short outage stays visible after it recovers instead of disappearing the moment the check passes green again.
• Maintenance windows and embeds. Planned maintenance is recorded as planned rather than as a failure, and the current state can be embedded elsewhere as a badge so the same data serves both the site and the community Discord.
• Deliberate separation. The status backend binds to loopback behind nginx and holds no credentials for the hosts it reports on. It can describe the fleet without being able to touch it.

Measurable Outcomes & Limitations

• Memory Efficiency: Telemetry collection daemon uses less than 15MB RAM per bare-metal host, representing over 90% resource reduction compared to off-the-shelf agents.
• Preemptive Resolution: Reduced average time-to-detect (TTD) from several minutes down to under 10 seconds, allowing engineers to intervene before players experienced disconnections.
• Zero Alert Storms: Implemented exponential backoff and grouping logic, eliminating false-positive notification floods.
• Honest Limitations: Relies on WebSocket connectivity to the central collector; during total upstream network loss on a host, heartbeats detect silence rather than receiving root-cause diagnostics until link restoration. Public traffic to the status page is subject to an edge bot challenge, so automated curl checks see a 403 even while the page itself is healthy.

Technologies Used

• Telemetry Core: Node.js, POSIX Shell (Bash), Linux Procfs
• Data Layer: Redis (Pub/Sub & Time-window caching), Node.js built-in SQLite (node:sqlite)
• Transport: WebSocket (ws), HTTPS, TLS Encryption
• Notification: Discord Webhook API (Rich Embed formatting)
• Status Page: Express, React + Vite front end, nginx reverse proxy, systemd units
• Process Management: Systemd Units, Systemd Timers