Production hardening

Phase 9 adds a three-layer rate limiter that composes with the existing replay defences. The cryptographic substrate (Bloom / sliding window / response chain) catches token reuse; the rate limiter catches floods of novel tokens that the cryptographic layer happily accepts as fresh.

The three layers

The asymmetric audit treatment is intentional: the two DDoS- prevention layers bypass the audit emitter (ring-buffer back- pressure under sustained flood is exactly the failure mode they prevent); the forensic layer emits because operators need “credential X is being abused” in the audit log, and the per- sub-ID throughput is by definition low enough that emitting per throttle isn’t a back-pressure surface.

Configuration

Each layer is configured independently via env. The boot default is unlimited() — short-circuits to LimitOutcome::Allowed with zero map allocation, so deployments don’t see a hot-path regression on upgrade.

Per-source-IP

Per-fleet (global default)

Per-fleet overrides land in the manifest (see below).

Per-(fleet, sub-ID)

Per-fleet manifest override

Manifest gains optional per-fleet rate-limit fields. Both must be set together (or both omitted); partial overrides are rejected at boot.

[[fleet]]
id                        = "prod-hft"
seed_path                 = "seeds/prod-hft.seed"
preset                    = "high-security"
rate_limit_burst          = 50       # tighter than the global default
rate_limit_refill_per_sec = 25.0

SIGHUP-reload applies the new policy on the next request. The bucket’s accumulated (tokens, last_refill) carry forward across the policy change — refill uses the new rate; tokens clamp to the new burst. This is the right semantics for live tuning: operators don’t want a “reset” on every policy bump.

XFF chain resolver

The per-source-IP limiter needs the real client IP, not the upstream proxy’s. In production the proxy sits behind one or more layers (Caddy / Cloudflare / a cloud LB); the TCP peer IP is the upstream’s, and the real client IP is in X-Forwarded-For.

Trust list

EDSSA_TRUSTED_PROXIES=127.0.0.1,10.0.0.0/8,2001:db8::/32,fc00::/7

Comma-separated mix of bare IPs and CIDR ranges (IPv4 + IPv6). Empty trust list (the boot default) disables XFF parsing entirely — operators behind a real upstream MUST set this; otherwise the limiter throttles by upstream IP only. This is the loud-failure-safe default: aggressive throttling on the upstream fires immediately and operators fix the env var.

Algorithm

1. Read X-Forwarded-For: ip1, ip2, ip3, ….
2. If trust list is empty → return peer IP (XFF parsing disabled).
3. If peer IP is not in the trust list → return peer IP. The request didn’t traverse a trusted proxy, so XFF is meaningless. Defends against an attacker speaking directly to the proxy with a spoofed XFF header.
4. Walk XFF rightmost-first; drop trusted hops; first untrusted entry is the client.
5. Malformed XFF entry → bail to peer IP. Attacker-injected garbage never silently uses a stale chain entry.
6. Every XFF entry trusted → log warn + return leftmost as best-effort (operator’s trust list covers the client’s network; the warn is observable).

Why rightmost-first walk is safe against spoofing

X-Forwarded-For is a hop-by-hop header set by upstream proxies. An attacker speaking directly to the proxy with X-Forwarded-For: 1.2.3.4 has the peer IP appended after the spoofed value. The rightmost-first walk reaches the peer (untrusted, since the attacker isn’t in the trust list) and returns peer — never the spoofed value.

The only way to forge a client IP is to be a trusted proxy yourself, which is the operator’s security boundary by definition.

Reject responses

All three layers return:

• HTTP 429 Too Many Requests (not 401).
• Retry-After: <seconds> header. Rounded up to 1 second minimum so clients don’t hot-loop. Capped at u32::MAX so the header value stays sane even for never-refill buckets (refill_per_sec = 0).
• Empty body (ADR-006).

The per-source-IP + per-fleet layers do NOT call the audit emitter; the per-(fleet, sub-ID) layer DOES (security-relevant). All three increment Prometheus counters regardless.

Observability

Counters

Gauges

Alert on edssa_ratelimit_key_count{scope=...} / edssa_ratelimit_max_keys{scope=...} > 0.8 to spot when an LRU cap is being approached. Fail-open semantics mean the limiter doesn’t block traffic at cap — but operators should bump the cap before the unobserved tail starts breaking isolation.

Device liveness

A device that fails stops presenting; it does not present badly. So a fleet’s accept rate stays at 100% while a member is dead — the remaining devices keep verifying — and no counter above moves. These gauges are the signal for that failure.

# A device that presented recently has gone silent.
edssa_devices_live < edssa_devices_seen        # for: 1m
# Device cardinality cap reached — per-device attribution is now
# rolled up per fleet, so `seen` understates the real device count.
edssa_device_registry_overflowed == 1

Two properties worth knowing before you alert on these:

• Exported only when the device registry is enabled (EDSSA_DEVICE_REGISTRY_DIR). The families are absent otherwise — absent means not measured, which is a different fact from zero. A fleet that has been observed and has since gone entirely quiet is published at 0, not dropped, so a dashboard cannot freeze at a stale count.
• Per-process, like every metric here. A proxy restart resets both gauges together, so the went-silent comparison cannot false-fire on a restart (0 < 0 is false) and repopulates within one live window as devices present again. Device identity is the claimed sub-ID of whoever holds the fleet seed — attested activity, not a hardware-rooted identity. The sub-ID is deliberately not a label (see the counter table).

Both windows are fixed in the server build. The 15-second live window is sized from measurement rather than taste: on the reference hardware fleet a healthy device’s staleness is sub-second (p50 0.27 s, p95 0.52 s over 3556 observations) while real outages last minutes, so nothing legitimate occupies the band between.

These gauges detect a silence a flush-interval sooner than the file-based device roster does, and the difference is data freshness, not disagreement. The gauges read the proxy’s live in-memory aggregate, so they turn over within the 15-second window. Anything reading the flushed JSONL registry is looking at a snapshot up to one EDSSA_DEVICE_FLUSH_INTERVAL_SECS old and widens its comparison by exactly that lag — so it never reports a healthy device as silent merely because the file has not been rewritten, at the cost of detecting a real silence up to one flush later. Alert on the gauges; read the registry for per-device history.

Existing DDoS fast-path

The pre-Phase-9 reject paths for malformed headers (missing-header / malformed-header / wrong-length) already bypass the audit emitter — they increment the edssa_rejects_total{reason} counter only. The new 429 rate- limit reject paths follow the same discipline. Operators auditing the hot path can confirm: the audit emitter (audit_sink.emit) is only reached after fleet resolution succeeds AND the rate-limit gates pass.

Operator runbook

Tuning checklist

1. Boot with unlimited (the default). Measure baseline traffic for a week.
2. Set per-source-IP first — the cheapest layer, easiest to over-tighten. Start at 5× peak observed QPS per IP; alert on edssa_source_ip_throttled_total rate > 0 for legitimate clients.
3. Set per-fleet next — global default; per-fleet manifest override for outliers. Tight bucket for HFT-grade fleets; loose for public-API fleets.
4. Set per-sub-ID last — the abuse-detection layer. Start permissive; tighten when audit-log shows sub-id-rate-limited rejects correlated with suspicious sub-IDs.
5. Configure EDSSA_TRUSTED_PROXIES the moment the proxy sits behind any upstream. Without it, per-source-IP throttles by upstream IP only (every client looks like one IP).

Suspected attack response

• edssa_source_ip_throttled_total{scope="source-ip"} spike alone → DDoS attempt; the per-source-IP layer is doing its job. Confirm peer IPs via tracing logs.
• edssa_ratelimit_throttled_total{fleet=...} spike → legitimate-looking traffic to one fleet. Check whether the fleet’s seed was rotated recently (legitimate spike) or if a credential has leaked.
• edssa_sub_id_throttled_total{fleet=...} + audit-log sub-id-rate-limited rejects → forensic signal that a specific credential is being abused. Rotate the seed (F-25 cooperative recovery) immediately; investigate the audit trail for the affected sub-ID.

What’s NOT in this batch

• Per-fleet rate-limit panel UI. Today operators edit fleets.toml directly and SIGHUP. A future batch lands a /fleets/:id/rate-limit form posting to a SIGHUP-trigger flow.
• Audit Tier-3 trace surfacing rate-limit state. The bucket’s (tokens, last_refill) snapshot at emit time would let forensic analysis replay the throttle decision.
• Criterion bench suite over the limiter hot path under simulated 10k QPS — operator capacity planning needs concrete numbers; lands as a focused perf-pass batch.