# Draft — source for SPEC/DR (rev. 2 — 2026-06-12)
# SPEC: docs/specs/20100612_1_scaling.md
# DR:  docs/DRs/20100612_1_scaling.md
#
The problem:
Now we have a small VM instance which serves not so much of things. A short list of services include:
- own web service (nginx+php) for UI/UX and some control panels, serving issues, tickets, graphs, build control (orchestration)
- git UI (gitea)
- database service - data store
- minimalistic OTA service for mobile app
- minimalistic WireGuard provider
- url shortener (nginx+php)
- build service (docker, orchestration)

The problem is, sooner or later we'll need to expand the amount of VM instances, delegate some features to separate virtual machines, most probably migrate to the clouds (GCP, Azure...), proper scaling with zero downtime with the ability of seamlessdeploy/update, add new services - and all this bundle should work ultra-stable, durable environments, splitting everything to the clusters of dev, staging, prod in a way that, for example, one deploys the unstable features to dev cluster, then the initial testing occurs, after decision making - move to the staging for beta tests, and finally to production once the decision is that staging behaves stable.

The how:
The proposed way is:
- one, let's say, cluster (let's call it dev cluster for example), which has N virtual machines, each of:
** web frontent VM(s) + OTA (linked together with common datastore VMs)
** database VM(s)
** data store VM(s)
** WireGuard VM(s)
** URL shortener VM + gitea for developers (1 small instance should be enough I believe)
** build service VM(s)
** N VMs for new services and features (starting from 0 and depend on the features)
- more tighter (money-wise) approach
- other
TODO: consult with AI for the others and the best practices of building scalable systems like proposed one depend on the current services and future needs

The simulation:
In theory, sysop (system operator aka project owner as for now) may simulate any (meaningful) number of HVMs for cluster simulation based on something lightweight like alpine linux distro plus sets of packages pre-installed.

The solution:
Ask AI to collect the information, make its decision based on all that we currently have plus look deeper into the future. The solution might be balanced between cost and providing services, for example, we don't need CPU or RAM resources for data storage but CPU/RAM is important for build service, etc.
Also, the matter of seamless scaling is super important, better to have the load balancing ready to use just from cloud provider.
Also, let AI think of how many simulatltaneous connections can we have per instance - the question is for RSSH implementation and future improvements

Network bandwidth (missing in rev.1 — add to SPEC/DR):
- we scale not only by CPU/RAM/connections but by **network throughput**: ingress/egress on FE WAN, FE→BE backhaul, per-VM NIC, CDN/object-store egress, SFU/ffmpeg media pipes, OTA/APK download storms, WireGuard UDP
- need **budget metrics** per service (Mbps sustained, peak, GB/month egress) and alerts before link saturation
- single FE uplink + single BE path is a hidden bottleneck today; cloud/hybrid phase must model **egress cost** (OTA, recordings, transcodes)

Global availability and GEO (missing in rev.1 — add to SPEC/DR):
- long-term: **the world can reach our services** (consoles, APIs, OTA, cast/signaling, short links, future commercial VPN) with acceptable latency — multi-region or CDN edge as we grow
- simultaneously: **segment isolation by policy** — not all content or features in all countries; example: certain recordings, OTA builds, or published cast archives must **not** be offered in Zambia (ZM) or other denylisted regions unless PO overrides
- enforcement should be layered: DNS/Geo routing, CDN/WAF geo rules, app/BE **geo_policy** on content objects, tenant RBAC — default allow globally, explicit **deny segments** per resource or catalog
- dev/staging may stay single-region; prod gains GEO rules when F3 cast storage / F5 payments / F9 streaming go live

The future key role requirements:
- multipoint casting and routing outside the intra network, usage of 3rdparty or own multipoint solutions on the backends(MCU, SFU...)
- new RBAC control, new roles - now we're thinking as developers and deliver the max of services to developers, but we will migrate to the other model sooner or later
- both realtime and non-realtime data streaming solutions, including streams re-compression (from any input to any output that ffmpeg can do for us, so ffmpeg will be involved)
- casts storage - we'll keep the records on the backends and provide them on demand if the user pays for it
- VPN capacities (question for all the resources), anonymous realtime VPN provider services (payed)
- payment systems integration (should cost near 0 for the developer's point of view of scaling/growth, it's more about marketing)
- potential payed add-on services (not decided yet on which side, on the backends or on mobiles) - user face masks, background extraction and replacement, etc.
- in-app money transfers between the app users (payed service)
- DNS control if possible - meaning selling the subdomains to the users, provide simplified own site builders accessible by own domains...
- 3rd party streaming providers integrations: realtime and non-realtime, let them stream and cast using our platform (I see ffmpeg active usage there plus cast handlers: low delay SFU-alike, mid-high delay using m3u playlists or similar hls-alike technologies)

AI should take everything described there plus the current project structure into account, weight everything, prioritize, use beautiful graphs and visual diagrams, and create SPEC/DR/ documents out of this draft using our common templates.
