Metadata-Version: 2.4
Name: welcome2-edge-agent
Version: 0.1.0
Summary: Welcome2 Proxmox/OPNsense edge agent — config sync, heartbeat, captive-portal callback
Author: Elcome Engineering
License: Proprietary
Requires-Python: <3.12,>=3.11
Description-Content-Type: text/markdown
Requires-Dist: fastapi<1.0,>=0.115
Requires-Dist: uvicorn[standard]<1.0,>=0.30
Requires-Dist: httpx<1.0,>=0.27
Requires-Dist: pydantic<3.0,>=2.6
Requires-Dist: pydantic-settings<3.0,>=2.2
Requires-Dist: structlog<25.0,>=24.1
Requires-Dist: tenacity<9.0,>=8.2
Requires-Dist: prometheus_client<1.0,>=0.20
Requires-Dist: dnspython<3.0,>=2.6
Provides-Extra: test
Requires-Dist: pytest<9.0,>=8.2; extra == "test"
Requires-Dist: pytest-asyncio<1.0,>=0.23; extra == "test"
Requires-Dist: respx<1.0,>=0.21; extra == "test"
Requires-Dist: freezegun<2.0,>=1.5; extra == "test"
Provides-Extra: dev
Requires-Dist: ruff<1.0,>=0.4; extra == "dev"
Requires-Dist: mypy<2.0,>=1.10; extra == "dev"

# welcome2-edge-agent

Python daemon that lives in a Proxmox LXC next to the OPNsense VM and
bridges Welcome2 cloud APIs to the OPNsense REST API.

## Responsibilities

| Task | Cadence | Where |
|------|---------|-------|
| Fetch site config from Welcome2 and push to OPNsense (aliases, captive-portal allowed addresses, template vars, local DNS records) | 60 s | `services/config_sync.py` |
| Resolve allowlisted hostnames via the guest-facing Unbound and union the answers (+ holdover) into the captive bypass; assert Unbound `cache-min-ttl` | Inline with config sync | `services/walled_garden_dns.py` |
| Heartbeat to Welcome2 with WAN/CP/RADIUS health | 60 s | `services/heartbeat.py` |
| Report OPNsense captive-portal session counters to Welcome2 metering | 5 s | `services/session_telemetry.py` |
| Report OPNsense interface, session, pre-auth, allowlist, NetFlow, DNS, category, and client-attribution telemetry | 5 s; low-priority details 120 s | `services/gateway_telemetry.py`, `services/host_telemetry.py` |
| Probe WAN/CP/RADIUS reachability | Inline with heartbeat | `services/health_probe.py` |
| Serve `GET /portal/start` renewal-aware entrypoint and `GET /portal/complete` browser callback | On demand | `routers/portal.py` |
| Resolve local DNS *shortcuts* (e.g. `dashboard`) by 302-redirecting to the site's tokenized URL | On demand | `routers/dns_redirect.py` |
| Consume session grants and drive OPNsense connect/disconnect (feature-flagged) | On demand | `services/session_control.py` |

## Architecture

```
FastAPI app
├── /health                                   — liveness
├── /portal/start?clientMac=…&redirurl=…      — local portal entry + renewal context
├── /portal/complete?client_mac=…&redirurl=…  — browser callback
├── /v1/session/start  (direct-session mode)
├── /v1/session/stop   (direct-session mode)
└── (catch-all, registered last) — local DNS shortcut redirects, keyed by Host

Background tasks (started from lifespan):
├── ConfigSyncService      — every EDGE_AGENT_CONFIG_SYNC_SECONDS
├── HeartbeatService       — every EDGE_AGENT_HEARTBEAT_SECONDS
├── SessionTelemetryService — every EDGE_AGENT_SESSION_TELEMETRY_SECONDS
└── GatewayTelemetryService — every EDGE_AGENT_GATEWAY_TELEMETRY_SECONDS

Clients:
├── Welcome2ApiClient      — httpx.AsyncClient, bearer auth
└── OPNsenseClient         — httpx.AsyncClient, basic auth (API key/secret)
```

Everything is a single process. We chose FastAPI because it's already in
the Welcome2 stack and gives us clean dependency injection for the
background tasks to share the two HTTP clients. Background tasks are
plain `asyncio.Task` objects managed in `lifespan` — not Celery, not
APScheduler. The workload is light enough that a bare asyncio loop is
the right tool.

## Install

The Proxmox bootstrap script at
[`../proxmox/edge-agent-lxc/bootstrap.sh`](../proxmox/edge-agent-lxc/bootstrap.sh)
handles this end-to-end. By hand:

```bash
uv sync --locked --extra test
uv run --locked welcome2-edge-agent --help
```

## Release packaging

Release artifacts are format-2 tarballs containing a locked Debian 12 /
CPython 3.11 / x86_64 wheelhouse, release-only offline uv configuration,
CycloneDX SBOM, and hashed manifest:

```bash
RELEASE_OUTPUT_DIR=release-artifacts scripts/build-release-artifact.sh HEAD
```

The builder uses pinned uv `0.11.22` in the target-compatible Docker image and
proves installation with an empty cache and networking disabled before sealing
the deterministic archive. To update dependencies, edit `pyproject.toml`,
refresh `uv.lock` with the pinned uv version, review the lock diff, rebuild and
review wheel/SBOM/manifest changes, then require exact-artifact development-fleet
qualification. The `release-build` group pins build backends needed in the
offline payload; it is not installed by runtime `uv sync --no-dev`.

## Run

```bash
# Copy the env template and edit the secrets
cp .env.example /etc/welcome2-edge-agent.env
chmod 0600 /etc/welcome2-edge-agent.env
vim /etc/welcome2-edge-agent.env

# Via systemd (preferred in the LXC)
systemctl start welcome2-edge-agent
journalctl -u welcome2-edge-agent -f

# Or directly (for local testing)
set -a && source /etc/welcome2-edge-agent.env && set +a
uv run --locked welcome2-edge-agent
```

## Environment variables

See [`.env.example`](.env.example) for the full list. Required:

- `WELCOME2_API_BASE_URL` — Welcome2 core-api URL (include APIM prefix).
- `WELCOME2_SITE_ID` — UUID of this site. **Optional**: if unset the agent
  boots in *unprovisioned* mode — it runs no site-bound loops (config-sync,
  heartbeat, telemetry, session control) and is ready to be adopted and
  assigned to a Welcome2 site. `/health` reports `"unprovisioned": true`.
  Set this to provision the gateway for a site.
- `WELCOME2_EDGE_BEARER_TOKEN` — per-site bearer from Welcome2. **Optional**
  in unprovisioned mode; **required** once `WELCOME2_SITE_ID` is set (the agent
  fails loud at startup if a site is configured without a token). During
  self-adoption it is minted by the backend and written automatically — see
  *Self-adoption* below.
- `GATEWAY_ENROLLMENT_TOKEN` — **optional** shared fleet enrollment token. When
  set on an unprovisioned gateway, the agent runs an announce poll-loop that
  self-adopts the gateway into a Welcome2 site (see *Self-adoption*). Leave
  unset to require manual provisioning.
- `OPNSENSE_API_URL` — e.g. `https://172.16.20.2` (OPNsense WAN on the SDN transit).
- `OPNSENSE_API_KEY` / `OPNSENSE_API_SECRET` — API credentials for the
  `welcome2-edge-agent` service user in OPNsense.
- `OPNSENSE_CAPTIVE_PORTAL_ZONE_ID` — OPNsense captive portal zone ID
  used for session and allowed-address sync.
- `GATEWAY_GUEST_MAC` — MAC of the OPNsense guest-side (VLAN 10) interface.
- `OPNSENSE_GUEST_NETWORK_CIDR` — guest/client subnet used to classify
  pre-auth and allowlist byte counters. The pilot default is `192.168.10.0/24`.
- `EDGE_AGENT_GATEWAY_TELEMETRY_SECONDS` — frequent gateway sample cadence.
  Default: 5 seconds.
- `EDGE_AGENT_GATEWAY_DETAIL_SECONDS` — low-priority detail refresh cadence for
  system inventory, interface inventory, local host sysfs/procfs, and LLDP
  neighbors. Default: 120 seconds.
- `EDGE_AGENT_CONNECTIVITY_HISTORY_PATH` — durable local WAN/internet
  continuity sample buffer. Samples are appended before upload and replayed on
  the next successful gateway telemetry post, so outages are preserved even
  while Welcome2 is unreachable.
- `EDGE_AGENT_CONNECTIVITY_PROBE_TARGETS` — comma-separated verification
  targets probed from OPNsense. Default: `1.1.1.1,8.8.8.8`.
- `EDGE_AGENT_CONNECTIVITY_UPLOAD_LIMIT` — max pending continuity samples sent
  in one telemetry payload. Default: 720.
- `EDGE_AGENT_HOST_SYS_PATH` — sysfs root used by low-rate host telemetry.
  New Proxmox LXCs bind-mount the host's `/sys` at `/mnt/host-sys`, which is
  the env-template default.
- `EDGE_AGENT_SENSORS_TELEMETRY_ENABLED` / `EDGE_AGENT_SENSORS_PATH` — optional
  `sensors -j` fallback parser used only when sysfs temperature sensors are not
  available in the edge-agent runtime.
- `EDGE_AGENT_FLOW_WINDOW_SECONDS` — Network Insight/NetFlow lookback window
  for top clients and services. Default: 300 seconds.
- `EDGE_AGENT_FLOW_TOP_LIMIT` / `EDGE_AGENT_DNS_TOP_LIMIT` — maximum rows
  sampled from OPNsense for NetFlow and DNS summaries. Default: 10.
- `EDGE_AGENT_RECENT_SESSION_TTL_SECONDS` — how long the local
  `/portal/start` entrypoint remembers a disconnected client and forwards
  `renewal=1` context to Welcome2. Default: 1800 seconds.

## Self-adoption (announce poll-loop)

An unprovisioned gateway can adopt itself into a Welcome2 site instead of
being provisioned by hand. When `GATEWAY_ENROLLMENT_TOKEN` is set and no
`WELCOME2_SITE_ID` is configured, the agent starts an announce poll-loop:

1. Every `EDGE_AGENT_ANNOUNCE_SECONDS` (default 60) it POSTs to
   `POST /v1/edge/announce` with `Authorization: Bearer
   <GATEWAY_ENROLLMENT_TOKEN>` and its `hardware_id`
   (`GATEWAY_HARDWARE_ID`, else `/etc/machine-id`, else hostname).
2. While the site has not adopted the gateway the backend replies
   `provisioned: false` and the agent keeps polling. `/health` surfaces the
   loop under `announce` (`enabled`, `hardware_id`, `last_announced_at`,
   `provisioned`, `last_error`).
3. When an operator adopts the gateway in Welcome2, the next announce returns
   `provisioned: true`, the `site_id`, and a freshly minted `edge_token`
   (delivered **exactly once**). The agent writes `WELCOME2_SITE_ID` and
   `WELCOME2_EDGE_BEARER_TOKEN` to the provisioning drop-in
   (`EDGE_AGENT_PROVISIONING_ENV_PATH`, default
   `/var/lib/welcome2-edge-agent/provisioning.env`, mode `0600`) and restarts
   itself (SIGTERM → systemd `Restart=always`) into provisioned mode.

The provisioning drop-in lives under the writable `StateDirectory` because
`ProtectSystem=strict` makes `/etc` read-only at runtime; the systemd unit
loads it via a second, optional `EnvironmentFile=-` that overrides the blank
base values in `/etc/welcome2-edge-agent.env`. Delete the drop-in (and clear
`WELCOME2_SITE_ID`) to return a gateway to unprovisioned mode.

Related settings: `EDGE_AGENT_ANNOUNCE_SECONDS`, `GATEWAY_HARDWARE_ID`,
`GATEWAY_HARDWARE_ID_PATH`, `EDGE_AGENT_PROVISIONING_ENV_PATH`,
`EDGE_AGENT_SELF_PROVISION_RESTART` (set `false` to write the drop-in but skip
the automatic restart — mainly for tests).

## OPNsense permissions required

Grant the `welcome2-edge-agent` OPNsense service user only these
page privileges. The seed `config.xml` already lists them:

- `page-firewall-aliases`, `page-firewall-aliases-edit`
- `page-services-captiveportal`, `page-services-captiveportal-connect`,
  `page-services-captiveportal-disconnect`
- `page-status-captiveportal`, `page-status-services`, `page-status-system`
- `page-services-dhcp-server` (legacy DHCP service compatibility)
- `page-dhcp-kea-v4` (Kea DHCPv4 subnets and custom options, including
  RFC 8910/CAPPORT option 114 and the guest-VLAN DHCP search domain)
- `page-diagnostics-networkinsight`, `page-diagnostics-netflow`
- `page-diagnostics-ping` (WAN/internet continuity verification probes)
- `page-interfaces-overview` (interface inventory/details; telemetry degrades
  gracefully if older OPNsense builds map this differently)
- `page-services-unbound` (now also used to **write** local DNS host
  overrides via `settings/{add,set,del}HostOverride`, not just read DNS stats)

Do **not** give it shell, webGUI, or page-all.

## Gateway telemetry sources

The gateway telemetry loop is additive: if NetFlow or Unbound statistics are
disabled on OPNsense, the agent still posts the rest of the sample and marks
the affected sections disabled or empty.

- NetFlow top-client, top-service, and per-client service samples use
  `/api/diagnostics/networkinsight/top/...` over the last
  `EDGE_AGENT_FLOW_WINDOW_SECONDS` seconds, filtered to the guest interface
  when OPNsense exposes the interface device name.
- DNS totals, top domains, blocked domains, DNS top clients, and category
  summaries come from OPNsense Unbound reporting. Query statistics must be
  enabled locally on the gateway.
- When `STARLINK_TELEMETRY_ENABLED=true`, read-only Starlink dish status is
  sampled from the local gRPC endpoint through `grpcurl` and included in the
  same gateway telemetry payload. Samples are stored locally before upload and
  replayed as `starlink_history.samples` when Welcome2 is reachable again; the
  latest sample is also exposed locally at `/network/status` and
  `/network/status.json` so guests can see dish state during an ISP outage.
- Client attribution uses the union of active captive-portal clients, NetFlow
  top clients, and Unbound top clients. Per-client service/category details
  require local NetFlow collection; DNS client totals require Unbound query
  statistics.
- Low-priority details are refreshed every `EDGE_AGENT_GATEWAY_DETAIL_SECONDS`
  seconds and attached to the normal gateway payload with `detail_observed_at`.
  They include OPNsense interface overview details plus optional local
  procfs/sysfs host telemetry: CPU model, core/thread counts, load averages,
  memory/disk usage, temperature sensors, power sensors (hwmon/RAPL fallback),
  and physical interface carrier/speed/duplex/counters.
  Proxmox-hosted deployments should expose the host sysfs read-only at
  `/mnt/host-sys` using `proxmox/edge-agent-lxc/create-lxc.sh` or the retrofit
  script `attach-host-telemetry.sh`.
- When `EDGE_AGENT_LLDP_TELEMETRY_ENABLED=true`, the host collector runs
  `lldpctl -f json` if present. Any directly connected LLDP neighbors are added
  under the matching host interface's `neighbors` list, which is the best source
  for "what AP/switch/device is connected to this physical port." If `lldpctl`
  is absent or times out, the sample simply omits neighbors.
- WAN/internet continuity is sampled every `EDGE_AGENT_GATEWAY_TELEMETRY_SECONDS`
  seconds using OPNsense gateway status plus ICMP probes to
  `EDGE_AGENT_CONNECTIVITY_PROBE_TARGETS`. Samples are stored locally before
  upload and sent as `connectivity.samples`, with `online`, `degraded`, or
  `offline` status for timeline rendering. When Starlink telemetry is enabled,
  each connectivity sample carries whether the dish was locally reachable plus
  the current dish online/state/latency/drop and obstruction fields so Welcome2
  can distinguish public-internet outages from a missing local dish path.
- Rolling 5-minute, 1-hour, and 24-hour windows are calculated by the
  Welcome2 backend from stored gateway samples. DNS window counts are deltas
  from OPNsense Unbound's cumulative counters inside each window.

Starlink telemetry is enabled by default, and deploy/bootstrap scripts install
`grpcurl` when available. To verify a site's dish path, run this from inside the
edge-agent LXC:

```bash
grpcurl -plaintext -d '{"get_status":{}}' \
  192.168.100.1:9200 SpaceX.API.Device.Device/Handle
```

If it does not, add the OPNsense route/firewall path to the dish management
address first. The default env knobs are in `.env.example`:
`STARLINK_TELEMETRY_ENABLED`, `STARLINK_GRPC_TARGET`,
`STARLINK_GRPCURL_PATH`, `STARLINK_GRPC_TIMEOUT_SECONDS`,
`EDGE_AGENT_STARLINK_HISTORY_PATH`, `EDGE_AGENT_STARLINK_RETENTION_HOURS`, and
`EDGE_AGENT_STARLINK_UPLOAD_LIMIT`.

## Local DNS records / shortcuts

Each OPNsense site can carry a list of **custom local DNS records**, served
from `GET /v1/edge/site-config` as `local_dns_records` under a
`local_dns_domain` (default `welcome.guest`). On every config-sync pass the
agent reconciles them into OPNsense Unbound host overrides (tagged
`welcome-managed`, so operator-created overrides are never touched) and pushes
`local_dns_domain` as the guest-VLAN DHCP search suffix (Kea), so a bare
`dashboard` resolves as well as the FQDN `dashboard.welcome.guest`.

Two record kinds:

- **`host`** — a plain A record. `target` is an IPv4 address; Unbound answers
  with it directly (e.g. `printer` → `192.168.30.50`).
- **`redirect`** — `target` is a URL. DNS can't carry a URL/token, so the
  override points the name at the edge-agent's guest IP and the
  `routers/dns_redirect.py` catch-all issues a `302` to `target`. The built-in
  `dashboard` shortcut uses this to land guests on the site's tokenized portal
  URL.

In addition to site-provided records, the edge-agent always manages the FQDN
`dashboard.welcome.online` as a local Unbound override on each gateway. That
name resolves to the local edge-agent IP and is redirected to the gateway's
`/portal/start` endpoint (which then forwards to the site+gateway-specific
Welcome2 portal URL).

Shortcuts are reached over **http** (a typed bare name has no TLS cert here);
we 302 to the https tokenized URL. Existing DHCP leases pick up the search
domain on renewal — the FQDN form works immediately.

## Testing

```bash
pytest
```

Tests use [`respx`](https://github.com/lundberg/respx) to stub out both
Welcome2 and OPNsense, so no real network is hit.

## Feature flag: `OPNSENSE_DIRECT_SESSION_CONTROL`

This flag is **required with no default** — `config.py` deliberately fails
loudly at startup if the env omits it, so a misconfigured deploy can't
silently fall into the wrong path (`.env.example` sets `true`, the
canonical shape for OPNsense gateways).

- **true** — v2 mode, the canonical live path on OPNsense gateways.
  `/portal/complete` exchanges the portal return for
  `POST /v1/edge/session-grant` on Welcome2, then calls OPNsense's
  `/api/captiveportal/session/connect` to open the session directly.
  RADIUS is bypassed for auth; metering and session lifecycle are posted
  through edge-agent session telemetry to Welcome2. Requires the matching
  flag on Welcome2's core-api (managed in welcome2-infra) — otherwise the
  session-grant endpoint returns 501.
- **false** — legacy v1 mode. OPNsense's own captive portal drives
  RADIUS against Welcome2 FreeRADIUS. The edge-agent's
  `/portal/complete` callback just forwards the access code from the
  browser back to OPNsense's captive-portal login form. Session
  grants are never requested.
