gw: implement PROXY protocol

[kvinwang]

Summary

Add PROXY protocol (v1/v2) support to dstack-gateway, with the decision of whether to send a PP header to a backend made per (instance, port) — not per-client, not per-gateway.

Background & security

An earlier revision of this PR encoded PP as a p suffix in the SNI subdomain (e.g. app-8080p.domain.com). That's client-controlled: a client could connect to a PP-expecting port without the suffix, the gateway would skip writing the PP header, and the backend would fall back to the raw TCP peer address — the gateway's WireGuard IP — as the source. Effectively a source-address spoof.

PP must therefore be declared by the app itself and delivered to the gateway through channels that clients cannot forge.

Design

1. AppCompose.ports (dstack-types)

Apps declare per-port attributes in app-compose.json:

{
  "ports": [{ "port": 8080, "pp": true }]
}

Because it's part of app-compose, the declaration is measured into compose_hash and covered by attestation.

2. Reported at registration (RegisterCvmRequest.port_attrs)

New CVMs include their port_attrs in the existing WireGuard registration RPC. The field is wrapped in optional PortAttrsList so the gateway can distinguish "not reported" (old dstack-util) from "reported empty" (new CVM with no PP-enabled port).

3. Stored per-instance, synced across gateway nodes

InstanceInfo/InstanceData grow a port_attrs map and a port_attrs_hash (the compose_hash it was learned against). Both are persisted in the existing WaveKV inst/{instance_id} record, so per-instance decisions survive gateway restarts and propagate across the cluster without extra keys.

Different instances of the same app may legitimately run different compose hashes (rolling upgrades), so caching is keyed by instance_id, not app_id.

4. Per-connection decision

AddressInfo carries the instance_id and connect_multiple_hosts returns the winner's id. should_send_pp(state, instance_id, port) consults the cached port_attrs:

• Hit → send PP header (or not) as declared.
• Miss → lazy-fetch via the agent's Info() RPC (see next section), cache, and use the result.
• Any failure → default pp=false.

5. Backward compatibility

Legacy CVMs that don't yet ship port_attrs at registration:

• Registration stores port_attrs: None but records the attested compose_hash.
• On the first proxied connection, the gateway calls http://{cvm_ip}:{agent_port}/prpc Info(), parses tcb_info.app_compose, extracts ports, and writes the result back to WaveKV.
• Subsequent connections hit the cache.

Subtleties:

• Re-registration with port_attrs=None does not wipe previously cached attrs (avoids a redundant lazy fetch every 3 minutes when dstack-util is old).
• Re-registration with a different compose_hash (app upgraded in place — typical for KMS-provisioned CVMs that reuse their disk) does invalidate the cache so stale PP flags don't outlive the upgrade.

6. Inbound PP

inbound_pp_enabled (server config) tells the gateway to read a PP header from the inbound TCP stream — used when the gateway sits behind a PP-aware LB like Cloudflare. When disabled, the gateway synthesises a PP header from the real TCP peer. Either way, the resulting header is what gets forwarded to the backend (when enabled).

Config surface

In gateway.toml under [core.proxy]:

agent_port = 8090          # Guest-agent port inside each CVM (used for Info() fetch)
inbound_pp_enabled = false # Read PP header from upstream (e.g. Cloudflare)

[core.proxy.timeouts]
pp_header = "5s"           # Timeout for reading inbound PP header

There is no global outbound_pp_enabled — per-port control comes from app-compose.json.

Files changed

• dstack-types/src/lib.rs — AppCompose.ports, PortAttrs
• gateway/rpc/proto/gateway_rpc.proto — PortAttrs, PortAttrsList, RegisterCvmRequest.port_attrs
• dstack-util/src/system_setup.rs — CVM reports port_attrs during registration
• gateway/src/pp.rs (new) — PP v1/v2 header parse + synthesis
• gateway/src/proxy/port_attrs.rs (new) — should_send_pp + lazy fetch
• gateway/src/{config,main_service,models,debug_service}.rs, gateway/src/kv/mod.rs, gateway/src/proxy/{proxy,tls_passthough,tls_terminate}.rs — wiring

Test plan

• cargo check --workspace
• cargo test -p dstack-gateway (8 tests pass, snapshots updated)
• cargo fmt --all
• Manual: deploy a CVM with ports: [{port: 8080, pp: true}], confirm PP header is received at the backend
• Manual: deploy a legacy CVM (no ports field), confirm no PP header is sent
• Manual: rolling-upgrade a KMS-provisioned CVM from pp: true to pp: false, confirm the cache is invalidated on the first re-registration with the new compose_hash
• Manual: behind Cloudflare with inbound_pp_enabled = true, confirm client IP is propagated end-to-end

[kvinwang]

End-to-end test on tdxlab

Deployed an nginx app with pp=true on port 8080 and pp=false on port 8081, exercised both forward and reverse paths.

Test endpoints

• 🟢 PP enabled: https://dc7f9c87d80e4bf47bc9dc7ab27f371cf9455144-8080.tdxlab.dstack.org:13004/
• 🔴 PP disabled: https://dc7f9c87d80e4bf47bc9dc7ab27f371cf9455144-8081.tdxlab.dstack.org:13004/

App ID a1586902ff3d0860ea024f3ebd87e91d508ed675, instance ID dc7f9c87d80e4bf47bc9dc7ab27f371cf9455144. CVM uses dstack-0.5.8 (legacy image — does not yet ship port_attrs at registration), so port_attrs is populated via the lazy Info() fetch path. Backward-compat path verified end to end.

Outbound PP (per-port, declared in app-compose.json)

{ "ports": [{"port": 8080, "pp": true}, {"port": 8081, "pp": false}] }

Port	pp	Backend sees
8080	true	proxy_protocol_addr=107.131.79.101 (real client)
8081	false	remote_addr=10.8.42.1 (gateway WG IP — client IP lost, as expected)

Inbound PP (gateway behind a PP-aware LB)

Set inbound_pp_enabled = true, moved gateway listen to :13006, fronted with haproxy on :13004 using send-proxy-v2:

client (107.131.79.101) → haproxy:13004 → [PP v2] → gateway:13006 → [PP v2] → backend

Result on pp=true port: origin addr = 107.131.79.101 — the real client IP propagates through both hops.

[Copilot]

Pull request overview

Adds end-to-end PROXY protocol (v1/v2) support in dstack-gateway, with per-(instance_id, port) outbound decisioning sourced from attested app-compose metadata (and lazily fetched for legacy CVMs), plus optional inbound PP parsing when the gateway is behind a PP-aware LB.

Changes:

• Introduce AppCompose.ports / PortAttrs and propagate port attributes through CVM registration (protobuf + dstack-util).
• Add gateway-side PP header read/synthesis and conditional outbound PP header injection per selected backend instance/port.
• Persist per-instance port attributes in WaveKV with compose-hash invalidation and legacy lazy fetch via guest-agent Info().

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
guest-agent/src/rpc_service.rs	Updates test fixture for new AppCompose.ports field.
gateway/src/proxy/tls_terminate.rs	Injects outbound PP header (when enabled) before bridging to backend.
gateway/src/proxy/tls_passthough.rs	Carries PP header through passthrough flow; returns winning instance_id from racing connect; injects PP header per port.
gateway/src/proxy/port_attrs.rs	New: per-instance/port lookup with legacy lazy fetch via agent Info().
gateway/src/proxy.rs	Reads/synthesizes inbound PP header before SNI extraction; passes header through proxy paths.
gateway/src/pp.rs	New: inbound PROXY protocol v1/v2 parse + synthesized header creation + display helper.
gateway/src/models.rs	Extends InstanceInfo with port_attrs and port_attrs_hash.
gateway/src/main_service/tests.rs	Adjusts test calls for updated registration/new_client signatures.
gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config.snap	Snapshot update for new InstanceInfo fields.
gateway/src/main_service/snapshots/dstack_gateway__main_service__tests__config-2.snap	Snapshot update for new InstanceInfo fields.
gateway/src/main_service.rs	Wires registration to store port attrs + compose hash; persists/invalidates cache; exposes instance lookup/update helpers; threads instance_id through address selection.
gateway/src/main.rs	Registers new pp module.
gateway/src/kv/mod.rs	Persists port_attrs and port_attrs_hash in InstanceData; defines PortFlags.
gateway/src/debug_service.rs	Updates debug registration call signature.
gateway/src/config.rs	Adds agent_port, inbound_pp_enabled, and timeouts.pp_header.
gateway/rpc/proto/gateway_rpc.proto	Adds PortAttrs / PortAttrsList; extends RegisterCvmRequest with optional port_attrs.
gateway/gateway.toml	Adds inbound_pp_enabled and pp_header timeout configuration.
gateway/Cargo.toml	Adds proxy-protocol dependency.
dstack-util/src/system_setup.rs	Sends port_attrs during registration based on app-compose ports.
dstack-types/src/lib.rs	Adds AppCompose.ports and PortAttrs schema.
Cargo.toml	Adds workspace dependency pin for proxy-protocol.
Cargo.lock	Locks new dependency graph for proxy-protocol and transitive deps.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.