Infrastructure health monitoring draft
milestone: R0-pre
severity: high
state: closed — DR/SPEC generated; see DRs/20260626_cluster_monitoring.md + specs/20260626_cluster_monitoring.md
       Track B (orchestration) → drafts/20260626_cluster_orchestration.txt
author: Anton Afanasyeu
date: 2026-06-26
related to: 20260618_repos_reorganizing*, microservice split, clustered CI/CD

The problem:
Actually there's no problem while everything is working. But as many know, sometimes there's a time where the software fails, hardware tires, and vice versa.
The goal is to do constant infrastructure health monitoring, probably even with prediction warnings, if possible


The why:
1) now we do have a lots of virtual machines (will be more if we move forward to real clouds) and lots of microservices splitted around the virtual machines. Sometimes the relation between machine and microservice is pretty straight forward:
| microservice A | -> | VM A | -> ... networking
sometimes more robust, but with additional cost of redundancy, like:
| microservice A | -> | VM B database  frontend |
                                |------> [ VM A master db node  ]----------> [aggregator + frontent]-----> ... networking
                                |------> [ VM C replica db node ]-----^
sometimes it may be very complex, for example
| microservice A (master) |
      |------------- [delegation A.A, slave ]
      |                   |-------------------[ VM A master db node ]-----|--------->[aggregator + frontend]------> ... next pipeline ...
      |                             |---------[ VM C replica db node ]----X----^
      |                                                                 | |
      |------------- [delegation A.B, slave ]                           | |
                          |-------------------[ VM A master db node ]----X----
                                    |---------[ VM B replica db node ]---|

[TRACK B — auto-remediation and auto-scaling scenarios moved to drafts/20260626_cluster_orchestration.txt]

Basically, the most important set of monitored events may be:
- hardware health monitoring
-- CPU: utilization %, steal time (Xen-critical), load average 1m/5m/15m
-- RAM: used / available / swap usage
-- Disk: utilization %, IOPS, write latency, SMART health
-- NIC: ingress/egress bytes/sec, packet drops, errors
-- Xen guest: vcpu steal, balloon driver state

- software health monitoring
-- node / microservice availability (process presence + HTTP health probe)
-- HTTP response time and status code per service endpoint
-- log error rate (ERROR/CRITICAL lines per minute, syslog/journald)
-- MariaDB: replication lag (seconds_behind_master), connection count, slow query count
-- nginx: 5xx rate, upstream connection errors
-- php-fpm: queue depth, worker saturation
-- cause classification: scale needed / bad config / process crash / OOM

- provider's fault
-- Xen host / uplink availability detected via external probe (UptimeRobot, no install)
-- internal: cast04 blackbox_exporter checks FE endpoint from inside cluster
-- software/service absence, payment expiry: ops calendar + manual review (not auto-detectable)

a simple example, using the current lab infrastructure:
[global net x.x.x.x] <-> [(134.17.26.161 external IP) router (debian-based] (internal IP 10.7.6.228) ] <-> [(10.7.0.10 intra network 1) FE (intra network 2: 10.7.0.0/8)] [intra network 3: ...]
the owner of top-level services is Anton Afanasyeu. top level services include:
- internet provider
- router maintenance
- intra network delegation, configuration, maintenance, VMs (empty, just a (re)source providing), git as a service, disk storage
- basic on-demand OS installation, on-demand VM snapshots
----------- all up is SaaS, for all the questions contact owner ---------

-- cluster of VM's, XEN-based (paravirt, Alpine latest LTS)
-- VM's control, VM's internal software installation, access, aggregation, fine tunings, software development, ssh, internal setups, etc.
----------- the last few lines are totally controllable by developer, developed product owner and projects workers as decided by product owner ---------

MONITORING PLACEMENT DECISION: Option A — cast04 (dedicated 4th Alpine VM, monitoring-only node).
Monitoring stack lives on cast04, isolated from the 3 workload nodes cast01/cast02/cast03.
Router and FE are config-only (no new software); monitoring uses cluster-internal network only.

External check for provider-fault: UptimeRobot free tier (web registration only, no install anywhere).


The how — DECIDED STACK:

INFRA METRICS (cast04):
  - prometheus + node_exporter on cast01/cast02/cast03/cast04 (apk add)
  - prometheus-blackbox-exporter on cast04 for HTTP/TCP service probes (apk add)
  - prometheus-mysqld-exporter on MariaDB node (apk add)
  - alertmanager on cast04 → email (SMTP relay) + Telegram bot (API only)
  - grafana on cast04 (apk add); community dashboards for node_exporter + MariaDB

UPTIME STATUS PAGE (cast04, Docker):
  - Uptime Kuma (Docker) on cast04 for a clean per-service status page
  - Docker itself available via apk

EXTERNAL AVAILABILITY:
  - UptimeRobot free tier — no install, web registration, HTTP probe to apps.f0xx.org
  - Alerting via email from UptimeRobot

NOT CHOSEN (and why):
  - Kibana/ELK: excluded (log-centric, heavy, wrong tool for infra metrics)
  - Zabbix: licensing OK but less flexible for future business metrics integration vs Grafana
  - Nagios: legacy, superseded by Prometheus+Alertmanager for this use case
  - VictoriaMetrics: not in Alpine apk repos; use standard Prometheus at this scale
  - Grafana Cloud / external SaaS aggregator: data must stay self-hosted (Anton's decision)

FUTURE (Track B, R1+):
  - Grafana Loki + Promtail for centralized log aggregation
  - Alertmanager webhook handlers for auto-remediation triggers
  - Grafana queries TimescaleDB for business/application metrics (complement, not replace Prometheus)

CONSTRAINTS (Alpine notes):
  - Machine: paravirtualized XEN guest (no HVM)
  - OS: Alpine latest LTS (community + edge repos)
  - Software: apk packages only; Docker available via apk for Uptime Kuma
  - FE/router: no software installation; config-only (iptables, routes, existing software)

PRE-REQUISITES (accounts to create before deploy):
  - Email alert relay: Gmail account + app password, OR Mailgun free tier (outbound SMTP only)
    NOTE: ImprovMX and addy.io are inbound email routing services — NOT usable as SMTP relay for Alertmanager
  - Telegram alerts: @BotFather → create bot → get TOKEN + CHAT_ID (add bot to ops group)
    No software install; Alertmanager calls Telegram API from cast04
  - UptimeRobot: free account at uptimerobot.com → add HTTP monitor for apps.f0xx.org

DNS / FE NGINX:
  - R0: no DNS changes, no FE nginx changes required
  - cast04 monitoring stack is internal-only (access via SSH tunnel or WireGuard from dev machine)
  - Future optional: FE nginx proxy rule (config-only) to expose Grafana at e.g. /monitoring/ externally
