docs(netbird): add N2 primary-deploy runbook (embedded Dex, B-refined cloudflared+Traefik, SQLite) (#258)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Backtalk6858
@@ -0,0 +1,173 @@
|
||||
# N2 — NetBird PRIMARY Production Deploy (owner-guided, main session) — #258
|
||||
|
||||
> **NOT a background agent.** N2's core is production mutations on primary + an interactive installer,
|
||||
> both of which a bg agent can't self-serve (see `playbook_background_agent_prompts` anti-pattern
|
||||
> 2026-09-18: don't spawn when the CORE deliverable needs a privilege the agent can't obtain). This is an
|
||||
> **owner-in-the-loop runbook** the main session drives step by step.
|
||||
>
|
||||
> **IdP = embedded Dex** (single-factor email/password, first admin at `/setup`). **Store = SQLite**
|
||||
> (decided 2026-09-18: 5 nodes = trivial scale; avoids coupling the mesh control plane to the shared,
|
||||
> OOM-prone Postgres on the RAM-oversubscribed primary; matches the proven sandbox path). N0's Postgres
|
||||
> `netbird` DB + `secret/netbird/db` DSN go unused — harmless, leave them.
|
||||
> **No cutover in N2** — NetBird runs ALONGSIDE cloudflared. Cutover = N3.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Architecture decision (settled here — the playbook's two-plane split was written for old NetBird)
|
||||
|
||||
NetBird v0.65+ = **one `netbirdio/netbird-server` container** (mgmt + signal + relay + STUN + embedded
|
||||
Dex on port 80) + a `netbird-dashboard` container. Vendor reverse-proxy routing (confirmed from
|
||||
docs.netbird.io/selfhosted/reverse-proxy, 2026-09-18):
|
||||
|
||||
| Path | Protocol | → | Why it matters |
|
||||
|---|---|---|---|
|
||||
| `/signalexchange.SignalExchange/*`, `/management.ManagementService/*`, `/management.ProxyService/*` | **gRPC (h2c)** | netbird-server:80 | control plane; proxy MUST do HTTP/2+gRPC |
|
||||
| `/relay*`, `/ws-proxy/*` | **WebSocket** | netbird-server:80 | **relay fallback now rides HTTPS**, not UDP |
|
||||
| `/api/*`, `/oauth2/*` | HTTP | netbird-server:80 | REST + embedded-Dex OIDC (issuer lives here) |
|
||||
| `/*` | HTTP | netbird-dashboard:80 | the SPA |
|
||||
| **UDP 3478 (STUN)** | UDP | netbird-server | **direct only — CANNOT be proxied**; enables direct p2p |
|
||||
|
||||
**Consequence:** the old "control plane via cloudflared / relay via direct UDP" split no longer maps —
|
||||
the relay is a WS path on the SAME origin as the control plane. The two realistic shapes:
|
||||
|
||||
**→ OPTION B-refined (DEFAULT, decided 2026-09-18): control plane via the existing cloudflared tunnel,
|
||||
open ONLY UDP for the mesh.** This is the settled N3 end-state — cloudflared ends up carrying essentially
|
||||
just the NetBird control-plane route while other services migrate onto the mesh; NetBird becomes the front
|
||||
door.
|
||||
- DNS: `netbird.reverseproxyserver.net` = a **cloudflared tunnel** hostname (proxied CNAME), exactly like
|
||||
the other services. gRPC + WebSocket both work over a cloudflared *tunnel* (the daemon, not CDN-proxy).
|
||||
- Gateway (owner): open **UDP 3478 (STUN)** only. Likely ALSO forward the **WireGuard UDP data port** to
|
||||
the primary for reliable direct p2p (confirm at Step 8 from `netbird status`; still UDP, no TCP).
|
||||
- No open inbound TCP; primary keeps its outbound-only tunnel posture.
|
||||
- `issuer == browser origin` still holds — the public hostname is the same whether via tunnel or direct.
|
||||
- **Relay tradeoff (accepted):** the rare relay *fallback* WS rides the cloudflared tunnel and may stutter
|
||||
for a NAT-hostile pair. With STUN (+ WG port) open, owner→own-primary should almost always be **direct
|
||||
p2p**, so the relay path is seldom hit. If a real device keeps landing on the relay and stuttering,
|
||||
upgrade that one path per Option A — with evidence, not on spec.
|
||||
- **Stealth note (accuracy):** UDP 3478 is a STUN *responder* → detectable/fingerprintable (low-risk:
|
||||
reflector only, no data plane). The genuinely-silent port is the **WireGuard data port** (WG drops
|
||||
unauthenticated packets with no reply). Encryption hides *contents*, not port existence.
|
||||
- **Hardening to apply:** firewall rate-limit on 3478 (STUN is a mild amplification vector); keep it
|
||||
**STUN-only, not an open TURN relay**; Dex auth + **manual peer approval** on enrollment; CF edge +
|
||||
Traefik protections on the control-plane HTTP plane behind the tunnel.
|
||||
|
||||
**OPTION A (documented upgrade only, if relay perf demands it): direct FQDN via existing Traefik.**
|
||||
- DNS: grey-cloud (DNS-only) A → primary public IP; gateway opens **TCP 443 → Traefik** + UDP 3478;
|
||||
Traefik TLS via DNS-01 wildcard (N0 CF token), gRPC service `scheme=h2c`.
|
||||
- Gets control plane AND relay on the direct path (no stutter) — but opens an inbound TCP listener and
|
||||
drops CF edge protection for that host. Only adopt with evidence a device needs it.
|
||||
|
||||
**DECISION GATE:** B-refined confirmed with owner 2026-09-18. Everything below assumes B-refined.
|
||||
|
||||
---
|
||||
|
||||
## Preconditions (verify, don't assume — read-only)
|
||||
- **CF DNS-01 token = NOT needed for N2 (B-refined).** TLS terminates at the CF edge via the tunnel, so no
|
||||
DNS-01 wildcard/`Zone:DNS:Edit` token is required to stand NetBird up. (That token is an **N3** dependency
|
||||
— needed only when services leave cloudflared for mesh-only and can't do http-01 through a dead tunnel.)
|
||||
- **N0-3 Vault paths** `secret/netbird/{management,setup-key}` exist (create if not). `oidc`/`turn`/`db`
|
||||
= N/A for embedded-Dex+SQLite.
|
||||
- **cloudflared** can add a new tunnel hostname → Traefik (owner has tunnel-config access). Confirm the
|
||||
tunnel passes gRPC + WebSocket (Step 5 watch-point).
|
||||
- **WireGuard on primary:** the routing-peer step needs the netbird client (kernel WG if the module is
|
||||
loaded, else userspace). Confirm before Step 8.
|
||||
|
||||
---
|
||||
|
||||
## Steps (main session issues each; owner runs prod mutations / browser / gateway)
|
||||
|
||||
### 0. Clean up the leftover sandbox (server-01) — do FIRST
|
||||
Owner runs (or paste-back): `ssh administrator@192.168.1.90` then
|
||||
`cd ~/netbird-edx-sandbox && docker compose down -v && rm -rf ~/netbird-edx-sandbox`.
|
||||
Verify `docker ps --format '{{.Names}}' | sort` == the 7-container baseline (agent-sudo, bitwarden-bridge-
|
||||
sandbox, hermes, jenkins, n8n-prod, n8n-sandbox, vault-sandbox). Not a blocker for primary, but don't
|
||||
leave cruft.
|
||||
|
||||
### 1. DNS + tunnel route (owner) — B-refined
|
||||
Add `netbird.reverseproxyserver.net` as a **cloudflared tunnel** hostname (proxied CNAME to the tunnel),
|
||||
same pattern as the other services. Tunnel ingress → the existing **Traefik** (which does the path/gRPC/WS
|
||||
splitting from the table above). No open TCP; no grey-cloud A-record. Verify the hostname resolves to the
|
||||
tunnel and `curl -I https://netbird.reverseproxyserver.net` reaches Traefik once Step 4 is up.
|
||||
> Watch-point: gRPC + WebSocket must survive cloudflared **tunnel** → Traefik → netbird-server (h2c).
|
||||
> Tunnels support both; confirm at Step 5.
|
||||
|
||||
### 2. Generate vendor artifacts (vendor-artifact-first — do NOT hand-write config)
|
||||
On primary, in a **scratch** dir (not the final compose dir yet):
|
||||
`curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started.sh -o getting-started.sh`
|
||||
Read it. Run it interactively with `NETBIRD_DOMAIN=netbird.reverseproxyserver.net`; when it asks for the
|
||||
reverse proxy, pick the option for **an existing/external reverse proxy** (NOT the bundled Traefik+LE).
|
||||
It emits `docker-compose.yml`, `config.yaml`, `dashboard.env` (+ `proxy.env` if the proxy service is
|
||||
enabled). **Read all of them.** Confirm from `config.yaml`: issuer/IdP = embedded Dex self-issued at
|
||||
`https://netbird.reverseproxyserver.net/...` (NOT an external URL); store/datadir = **SQLite** (a file
|
||||
under the data volume, not a Postgres DSN). Mask any secrets the script generated.
|
||||
|
||||
### 3. Adapt into the real location (main session writes; scope-limited)
|
||||
Move/adapt the generated files into **`/opt/appdata/docker/docker-compose/netbird/`** (the ONLY dir N2
|
||||
may create; touch no other service's compose):
|
||||
- **Remove the bundled Traefik/LE** service; keep only `netbird-server` + `netbird-dashboard`.
|
||||
- Attach both to the coolify docker network so the **existing Traefik** can route to them; bind container
|
||||
ports to `127.0.0.1` only (no direct external exposure except UDP 3478, which publishes `3478:3478/udp`).
|
||||
- **Existing-Traefik labels** per the table above — the gRPC router needs a service with
|
||||
`loadbalancer.server.scheme=h2c`; long timeouts for gRPC keep-alive; WS upgrade preserved on `/relay*`
|
||||
+ `/ws-proxy/*`. TLS is terminated at the CF edge via the tunnel (B-refined); Traefik routes internally
|
||||
as it does for the other tunneled services. (Option A instead terminates TLS at Traefik via the DNS-01
|
||||
wildcard resolver.)
|
||||
- Confirm `NETBIRD_DOMAIN` / dashboard `NETBIRD_MGMT_*` / OIDC authority all == `https://netbird.reverseproxyserver.net`
|
||||
(issuer == browser origin — the sandbox's "Unauthenticated" caveat vanishes only when this is exact).
|
||||
- **SQLite** store on a named volume under `/opt/appdata/docker/docker-compose/netbird/data`.
|
||||
- **Digest-pin** images; `restart: always`. Secrets (management secret, any relay/proxy token) via
|
||||
SecretSpec+Vault (`playbook_secretspec_env_resolution`) → write generated secrets to Vault
|
||||
`secret/netbird/management` etc., reference by env, never inline. Mask everywhere.
|
||||
|
||||
### 4. Bring it up (owner runs the mutation)
|
||||
Owner: `cd /opt/appdata/docker/docker-compose/netbird && docker compose up -d` (NEVER restart other
|
||||
containers — `feedback_no_docker_restarts`). Verify (separate command): `netbird-server` + `netbird-dashboard`
|
||||
Up/healthy; no resident container changed state.
|
||||
|
||||
### 5. Gateway + routing verify — B-refined (UDP only)
|
||||
- Owner opens **UDP 3478 (STUN)** on the gateway → netbird-server. Rate-limit it; keep STUN-only (no open
|
||||
TURN). Control plane needs **no** open TCP — it rides the cloudflared tunnel.
|
||||
- (WG data port: leave closed for now; revisit at Step 8 only if direct p2p to primary is unreliable —
|
||||
then forward that **UDP** port too.)
|
||||
- `curl -ksS https://netbird.reverseproxyserver.net/` serves the dashboard (via tunnel→Traefik); `.../api/`
|
||||
responds; OIDC discovery (`/.well-known/openid-configuration` under the Dex path) = 200 with issuer ==
|
||||
the origin. Confirm gRPC/WS survived the tunnel (management/signal reachable; a peer can enroll).
|
||||
|
||||
### 6. First admin (owner browser — the exact step Authelia hung on; MUST be clean here)
|
||||
Owner opens `https://netbird.reverseproxyserver.net/` → redirected to `/setup` → create admin
|
||||
(email / name / password) → **Create Account** → lands in the dashboard. This is the go/no-go that the
|
||||
sandbox already proved works with embedded Dex.
|
||||
|
||||
### 7. Setup key + enroll all 5 nodes (owner installs client per device)
|
||||
- Dashboard → Setup Keys → create a reusable key (**MASK it** everywhere).
|
||||
- Enroll the 5 nodes (owner runs `netbird up` w/ the key per device). Apply the **identity/ACL matrix**
|
||||
from design §2 (admin=laptop; daily=phone+tablet; servers group; phone-only SSH→primary grant; manual
|
||||
peer approval). **Routing peer on primary** advertises `192.168.1.0/24` + `172.16.16.0/24`; NetBird
|
||||
split-DNS `*.reverseproxyserver.net` → primary LAN/mesh IP via the routing peer.
|
||||
|
||||
### 8. Prove connectivity (folds in the N1 leftover)
|
||||
- **Direct p2p:** on a sustained transfer, `netbird status --detail` shows the pair **P2P/direct** (STUN
|
||||
did its job) — this is what keeps Jellyfin off the relay.
|
||||
- **Forced relay capture:** block the direct path between two peers and confirm `netbird status --detail`
|
||||
= **relayed via the NetBird relay** (the `/relay` WS on the direct FQDN), **not cloudflare**. Record
|
||||
`relay_transport` + `relay_is_cloudflare=false`.
|
||||
|
||||
### 9. Self-heal timer — WRITTEN, NOT registered
|
||||
Write the relay/health self-heal systemd unit but **do not register** it — defer to agent-sudo #176
|
||||
(same pattern as the Twingate connector). `restart: always` covers reboots meanwhile.
|
||||
|
||||
---
|
||||
|
||||
## Verify before calling N2 done
|
||||
All 5 nodes online; laptop reaches an admin surface over-mesh; phone reaches nextcloud + SSHes to primary
|
||||
over-mesh; a stream runs **direct** (not relayed); forced-relay case relays via NetBird (not cloudflare);
|
||||
issuer == origin (no "Unauthenticated"). Then **N2 = GREEN → N3 cutover** (staged, soak-gated).
|
||||
|
||||
## Persistence (main session, end-session checklist — NOT mid-run)
|
||||
Update `context.md` (primary + infra hubs), `playbook_netbird_phases` (fold N2 verdict; mark buildplan
|
||||
(b)/(c) resolved), design doc §7, `session_handoff`. Commit + push. Then N3.
|
||||
|
||||
## Wrap fields to capture
|
||||
`{nodes_enrolled, acl_applied, direct_stream_proven, relay_transport, relay_via:"tunnel|direct",
|
||||
issuer_matches_origin:true, timer_registered:false, gateway_ports_opened:[3478/udp (+wg/udp if needed)],
|
||||
notes}`
|
||||
Reference in New Issue
Block a user