Modern WAF internals: architecture, evasion, and NyxR production data

A Web Application Firewall (WAF) analyzes Hypertext Transfer Protocol (HTTP) traffic before it reaches the origin server, meaning the final application service. A modern WAF is stateful: it retains state across events to correlate a request with a client, policy, and history. It must also canonicalize each message, reducing equivalent representations to one common form exactly as downstream components do. If the proxy, an intermediary that receives and relays the request, the WAF, and the application assign different meanings to the same bytes, the detection engine no longer protects the message consumed by the origin.

On June 23, 2026, from 19:35 to 19:40 Paris time (UTC+2), the NyxR security gateway recorded 3,119 requests from one scanner on my application nyxr.app. In five minutes, it traversed 2,100 distinct Uniform Resource Identifiers (URIs), the resource identifiers targeted by requests. The data plane, the components that synchronously interpret and decide each request before a response can be sent, blocked 3,117 of them, or 99.94%, with 0 ms latency at the 50th percentile or median (p50) and the 95th percentile (p95). In parallel, ModSecurity, an open-source WAF engine, produced 3,121 audit transactions associated with rule 913100 from the Open Worldwide Application Security Project Core Rule Set (OWASP CRS), a generic ruleset that identifies the User-Agent, the string through which a client declares its software, of a known scanner here.

Recognizing a known scanner is only the visible part of the result. For such a block to be reliable, a modern WAF must also reconstruct the correct client address behind proxies, reject ambiguous HTTP messages, execute controls in the right order, apply policy locally, and record the decision without counting the same transaction twice. A mistake in any layer can create a bypass, a request that escapes the control, or a false positive, legitimate traffic incorrectly classified as malicious.

I designed NyxR around this chain. This article presents the resulting architecture choices from message framing to the final decision, then tests their behavior against one month of metrics produced by my application nyxr.app.

Architecture and the trust chain

The historical WAF model is a filter: a request enters, expressions compare it with signatures, and the request is accepted or rejected. That model explains a rule engine, but not a modern system.

A real request crosses several interpreters. The client serializes a message. A Content Delivery Network (CDN), a distributed network of caches and proxies, may receive it first; a load balancer distributes traffic across servers, while a reverse proxy relays requests on behalf of the origin. Each can decode and normalize the request, translate HTTP/2 to HTTP/1.1, add fields, and reuse a connection to the backend, the downstream service.

The WAF then builds its own variable collections. Finally, the application framework decodes the Uniform Resource Locator (URL), the address of the requested resource, along with cookies, forms, and structured bodies. JavaScript Object Notation (JSON) is a textual structured-data format; Extensible Markup Language (XML) is a markup language; a multipart body separates several parts, notably fields and files, with boundaries. Security depends on all these interpreters agreeing.

A Request for Comments (RFC) is a technical specification published within the Internet Engineering Task Force (IETF) ecosystem, the organization that standardizes Internet protocols. RFC 9110 separates HTTP semantics from its representations. RFC 9112 defines HTTP/1.1 framing and warns that divergent interpretations enable request smuggling, an attack that causes two successive components to delimit the same request stream differently. HTTP/2 and HTTP/3 move framing into binary frames, but conversion to an HTTP/1.1 backend creates a new interpretation boundary.

A modern WAF must therefore answer five distinct questions:

1. Where does the request begin and end? This is the framing and parsing problem, meaning structural analysis of the message.
2. Who or what probably sent it? This is the identity and proxy-trust problem.
3. What does its content mean after transformation? This is the canonicalization and detection problem.
4. What does its relation to earlier requests reveal? This is the temporal and behavioral problem.
5. Which action remains safe under failure? This is the architecture and degraded-mode problem.

Detection is only one step. Parsing, normalization, enrichment, decision, enforcement, the effective application of an action, logging, and learning need separate semantics.

These responsibilities have neither the same deadline nor the same failure mode. Organizing them into distinct planes keeps the current decision local while policy and analysis evolve on longer cycles.

Three planes, not a monolith. In a modern WAF architecture, a plane is a logical group of components organized around one responsibility and execution timing; it does not necessarily map to a separate machine. The data plane makes the current decision: it terminates the connection, establishes client context, applies local controls, inspects content, then blocks or forwards the request. The control plane manages policy publication. It receives the desired state, a declarative description of the policy expected by the operator, validates and versions it, then turns it into executable configuration without participating in the current request. The observation plane receives a copy of decisions after enforcement, retains history, correlates events, and proposes changes. A proposal can influence a future request only after passing back through the control plane; it never rewrites a decision already made. I designed NyxR around this separation as a self-hosted Unified Web Security Gateway that combines a reverse proxy, WAF, and behavioral controls in front of multiple services.

The critical path, or hot path, is the part of the data plane containing every operation a request must await before it receives a response. A database or remote service placed on that path therefore becomes a direct latency and availability dependency. In the NyxR implementation, the hot path combines NGINX, a web server and reverse proxy, OpenResty, an NGINX distribution that embeds the Lua scripting language through LuaJIT, a just-in-time compiler for Lua, then libModSecurity v3 and OWASP CRS. Transport Layer Security (TLS) encrypts a connection between two peers; its Server Name Indication (SNI) extension announces the target hostname during negotiation so the correct certificate and service can be selected.

Off that path, the control plane writes desired state to PostgreSQL, a relational database, while TypeScript, a typed variant of JavaScript, builds a bundle, a complete configuration package ready for activation. The observation plane transports events through Redis Streams, an ordered in-memory log provided by Redis, then a scorer turns weighted signals into a score. A rule in shadow mode, or observation mode, computes and logs the decision it would have made without changing the response sent to the client. Cyber Threat Intelligence (CTI) indicators are known threat data, such as addresses associated with malicious infrastructure.

This separation addresses a fundamental constraint: a synchronous request decision must not call PostgreSQL, an artificial-intelligence model, or an analytical datastore, storage optimized for aggregating and querying events, on every request. An Access Control List (ACL) is a set of rules that allows or denies traffic according to explicit criteria. An Internet Protocol (IP) address identifies an interface on an IP network; Classless Inter-Domain Routing (CIDR) notation combines an address with a prefix length to identify an entire network, for example 192.0.2.0/24. In a modern WAF, ACLs needed for the current request must be evaluated locally in the hot path so traffic can be blocked or passed without a remote-database dependency. A snapshot is a coherent copy frozen at a point in time. NyxR applies this principle to ACLs, reputation snapshots, counters, and other decision policies. A control-plane failure therefore leaves the last valid bundle serving traffic.

This local autonomy is real only if policy can be published without exposing a partial configuration. The activation mechanism must therefore preserve a last-known-good state and make each change verifiable before it reaches the data plane.

The config builder, the component that compiles desired state into executable files, does not edit the active file line by line. It renders a complete bundle, performs structural checks and nginx -t, and computes a checksum that detects any byte-level difference. It writes a new version and atomically swaps the active link, exposing no intermediate state, then reloads the gateway and performs a health check. An unhealthy activation restores the previous version.

For a penetration tester, this architecture provides three useful guarantees to verify:

• the policy version that served a request can be tied to the decision;
• control-plane compromise does not imply immediate data-plane compromise;
• an unavailable analytical dependency should not silently change the accepted HTTP grammar.

From socket to decision

Plane separation establishes who publishes policy, who decides, and who observes. It does not yet show the order in which the data plane acquires the information required for a decision, so the next path follows one request from the raw connection to the final action.

A network socket is the software endpoint through which the server receives a connection. From that socket onward, a request should not be modeled as a string passed to an is_malicious() function: it evolves as new information becomes available. A client fingerprint summarizes technical traits without claiming to identify the client; rate limiting bounds requests within a time window; a challenge requires additional proof, often browser execution, before processing continues.

Building the decision context

1. TLS termination and HTTP framing.

TLS termination decrypts the connection at the gateway and exposes the HTTP message. The ClientHello is the first message sent by the client during TLS negotiation; its versions, cipher suites, and extensions expose traits of the network stack. Termination also sets a trust boundary: an attacker who can reach the origin directly bypasses every control at the edge, the perimeter layer exposed to the Internet. Mutual TLS (mTLS) authenticates both server and client with certificates; together with network restriction or an origin secret, it prevents an unauthorized client from bypassing the gateway.

Before inspecting a payload, the useful content carried by the request, the gateway must reject messages whose length or structure could be interpreted differently downstream. HTTP Desync Attacks, The HTTP Garden, and WAFFLED show that parser disagreement is not an academic edge case. WAFFLED reports 1,207 confirmed bypasses across five widely deployed WAFs by mutating non-malicious aspects of JSON, XML, and multipart messages.

A modern WAF must reject framing ambiguity before trust decisions. In NyxR, I placed a local desynchronization guard at that exact point. Its job is not to detect Structured Query Language (SQL) injection, where input changes the structure of a database query, but to reject a request before a proxy and origin can assign two meanings to the same byte stream.

Framing establishes an unambiguous request representation, but not the network identity to which the decision should be attached. Once the message is delimited, the trust chain must determine which address may feed reputation, quotas, and bans.

2. Real client IP and the trust chain.

The X-Forwarded-For (XFF) header carries the address chain declared by traversed proxies; it is not an identity, but a claim. NGINX ngx_http_realip_module turns that claim into a client address only after trusted sources are configured through set_real_ip_from. If the whole Internet is trusted, the attacker selects their reputation, country, rate-limit key, and sometimes their allowlist status, meaning membership in an explicit list of permitted sources.

The correct invariant for a modern WAF is: only an explicitly trusted network hop may replace the socket peer address. In NyxR, I start with an empty trust set, publish approved proxy CIDRs in the bundle, and apply an additional guard. IPv4 and IPv6 are the two deployed versions of IP, with different address spaces and sometimes different routes. The PROXY protocol carries the original connection address outside HTTP headers between two mutually trusted proxies. A red-team assessment should test all these paths, plus direct access, the CDN, load balancer, and administration network.

A sufficiently trustworthy network identity makes address-keyed decisions usable. The data plane can then apply the cheapest local controls before reserving deep inspection for the requests that remain.

3. Cheap local gates.

ACLs, geolocation, CTI indicators, active bans, and selected reputation checks can run before deep body inspection. This saves processor time, memory, and origin bandwidth. The priority is safe only when bypass semantics are explicit: an allowlist must state which controls it skips and which remain mandatory.

These gates remove already-known sources or contexts, but an isolated request can still look ordinary. Behavioral correlation adds time, errors, and route sequence so automation spread across several transactions can be recognized.

4. Behavior and client fingerprint.

IP, the Autonomous System Number (ASN) of the network announcing that address on the Internet, the User-Agent family, header order, HTTP version, challenge outcomes, rate, HTTP 404 responses, authentication failures, and route sequences all have different lifetimes. Correlation can expose a distributed bot whose individual payloads look harmless.

Behavior describes how a client acts over time, not what the current transaction contains. ModSecurity and CRS provide that message-level inspection by searching the normalized request’s variables, headers, and body.

5. ModSecurity and CRS inspection.

The WAF parses available variables, applies deterministic transformations, executes rules, and accumulates an anomaly score. This score is the sum of points added by rules matching the transaction, not an attack probability. Body size, JSON depth, and argument-count limits are part of the control: an ignored parse error creates a region that the application understands but the WAF does not.

Inspection produces matches and a score, but it does not select the client-visible effect by itself. The final stage evaluates those signals under policy and chooses forwarding, rate limiting, challenge, temporary ban, or denial.

6. Friction and final decision.

Blocking is not always the best response. A challenge can distinguish an interactive browser from automation. A rate limit can protect an expensive endpoint, a combination of HTTP method and application route, without pretending that every request is malicious. A temporary ban can dampen a sequence. The final log must keep the applied action separate from the candidate reason.

This path places the rule engine among several independent controls. Interpreting its output correctly still requires understanding when a rule runs, which variables it can see, and how its points may become a disruptive action.

Interpreting ModSecurity and OWASP CRS

ModSecurity v3 executes rules in five documented phases: request headers, request body, response headers, response body, and logging. Data is cumulative. A phase 1 rule does not yet see every body argument; a phase 5 rule can no longer block the transaction. The ModSecurity v3 reference describes these contracts and v2 compatibility limitations.

The families shown below include Local File Inclusion (LFI), inclusion of a local file, Remote Code Execution (RCE), execution of remote code, Cross-Site Scripting (XSS), injection of code executed in a browser, and SQL injection (SQLi). The Paranoia Level (PL) selects the sensitivity of enabled rules. DetectionOnly mode retains detections without applying disruptive actions, while On permits actions such as rejecting the request.

Phases determine which data each rule can access, but not how several matches are aggregated. Anomaly scoring provides that composition layer before the blocking threshold is evaluated.

Anomaly score is not probability.

CRS uses collaborative detection. A critical rule typically adds 5 points, an error 4, a warning 3, and a notice 2. Detection rules are separate from the blocking evaluation. The Anomaly Scoring documentation explains how REQUEST-949-BLOCKING-EVALUATION.conf compares the total inbound score with the configured threshold.

A score of 10 means neither 10% risk nor twice the attack likelihood of score 5. It means rules added ten policy points. Defensible evidence retains at least:

• every matching rule identifier;
• examined variables and transformations;
• executed paranoia level;
• anomaly threshold;
• DetectionOnly or On mode;
• final gateway action.

The score explains how rules contribute to a transaction, but its sensitivity still depends on which rules are active and in what context. Paranoia level broadens coverage, while exclusions precisely remove legitimate targets that produce false positives.

Paranoia level and exclusions.

Paranoia levels add increasingly aggressive rules from PL1 to PL4. Raising the threshold to compensate for a higher PL can hide a single critical rule. A stronger strategy selects PL by risk, keeps a low threshold, observes false positives, and writes narrow exclusions.

A safe exclusion targets a rule, a URI, and ideally one parameter. Disabling the whole SQLi family because a Markdown field contains the word select fixes the false positive by removing protection. The following generic example does not reproduce NyxR configuration: REQUEST_URI is the received route, ARGS is the parameter collection, and REQBODY_ERROR indicates a body-parser failure.

1
# Remove only the "content" parameter from this rule on this endpoint.
2
SecRule REQUEST_URI "@beginsWith /api/articles" \
3
  "id:100100,phase:1,pass,nolog,ctl:ruleRemoveTargetById=942100;ARGS:content"
4

5
# Keep a parsing error as a blocking event.
6
SecRule REQBODY_ERROR "!@eq 0" \
7
  "id:100110,phase:2,t:none,log,deny,status:400,msg:'Request body parsing failed'"
8

9
# Publish the transaction scoring policy explicitly.
10
SecAction "id:100120,phase:1,pass,nolog,t:none,\
11
  setvar:tx.blocking_paranoia_level=2,\
12
  setvar:tx.inbound_anomaly_score_threshold=5"

Identifiers 100100 through 100120 are reserved for the example. Production identifiers need a documented local range that cannot collide with CRS.

Exclusions tune the scope of variables the engine can analyze; they provide no visibility into a truncated, oversized, or unparsable body. Resource bounds and the policy applied when they are exceeded therefore form a separate security boundary.

In the production extract I built, 12 transactions from three addresses triggered the local parsing-error control. The volume is small, but its semantics matter: NyxR distinguished “unrecognized content” from “recognized and benign content.”

Correlating identity without slowing the data plane

The preceding chain can decide one transaction, but its IP address cannot reliably connect several requests to the same software or behavior. Correlation must therefore enrich identity with protocol and temporal traits without turning the hot path into an analytical dependency.

Fingerprint and behavior

A Network Address Translation (NAT) gateway makes several clients share one public IP address; proxies and cloud egress create a similar effect. An IP address is therefore an observed network point, not a human identity.

Before sending the first HTTP byte, a TLS client emits a ClientHello. This message proposes TLS versions, cipher suites, signature algorithms, key groups, and extensions such as SNI or Application-Layer Protocol Negotiation (ALPN), which selects protocols such as HTTP/1.1 or HTTP/2; the server then selects compatible parameters in its response. The gateway terminating TLS therefore sees a structured description of the client library’s capabilities and choices before it sees the application request.

JA4 turns that description into an a_b_c fingerprint documented by the FoxIO project. The human-readable a section encodes transport type, TLS version, presence of an SNI hostname, cipher and extension counts, and an ALPN summary. Section b contains the first 12 hexadecimal characters of a hash over the sorted cipher-suite list. Section c does the same for sorted extensions, excluding SNI and ALPN, completed with signature algorithms. Secure Hash Algorithm 256-bit (SHA-256) is a hash function that reduces a variable-length list to a deterministic digest; JA4 retains only 12 characters for compactness.

Sorting removes the effect of randomized cipher and extension order. Generate Random Extensions And Sustain Extensibility (GREASE) values, synthetic values injected to keep TLS implementations from ossifying around a known list, are ignored. JA4 is therefore more stable than raw concatenation, but it is neither authentication nor a unique identifier: a shared library produces the same fingerprint across many hosts, a configurable stack can imitate a browser, an update can change the fingerprint, and upstream TLS termination can hide the original ClientHello.

A modern WAF can complement JA4 with HTTP traits instead of depending on one identifier. In NyxR, I chose a composite built from header order, User-Agent family, HTTP version, missing traits, and ALPN. A coherence measure compares client claims with protocol shape. The objective is not to prove maliciousness, but to make rotation of one attribute less effective.

A fingerprint stabilizes the client’s technical context, but it proves no malicious intent. Behavioral scoring adds time by accumulating only explainable events, then reducing their weight when they stop recurring.

The server-side behavioral score accumulates evidence and decays by half-life, the time after which a signal’s weight is divided by two. Only events with a positive signal touch Redis. A benign request does not artificially subtract points; silence decays the score. ModSecurity and access-log signals are disjoint so the same transaction is not scored twice.

The score applies deterministic rules that have already been published; it cannot discover new profiles or validate their safety by itself. Learning can analyze history and propose an evolution, provided it remains asynchronous and passes back through the control plane before enforcement.

Learning off the critical path

An unsafe adaptive system applies a probabilistic remote output to each request. It introduces network dependency, non-deterministic latency, provider quota exposure, and a security-mode change when the model is unavailable.

A modern adaptive architecture must separate the local scenario engine, which recognizes deterministic sequences, the asynchronous scorer, which aggregates events after the request, and the verdict service, which adds a classification opinion. I implemented that separation in NyxR: a learned rule starts in observation mode, where its result is logged without influencing the response, then the new policy passes through the build and validation pipeline before it can become active.

In NyxR, I designed the verdict service as asynchronous enrichment that never owns the current decision. The nyxr.app data provide a concrete example: during the final 24 hours of the window, this service exceeded its response deadline, the scorer logged fallback to the local threshold, and the gateway remained healthy. The useful fact is therefore not whether the artificial-intelligence model is available, but which deterministic decision remains active when it does not answer.

Across 482 durable bans created during the window:

• 275 came from the behavioral scorer and 207 from edge scenarios adopted into history;
• they covered 316 distinct targets;
• median score at ban was 42, the 95th percentile 103, and the maximum 4,224;
• stored enrichment concluded ban for 434, monitor for 46, and allow for 2.

Those two allow verdicts do not make every other decision wrong. They prove that a classifier, a component that assigns an observation to a category, and a deterministic threshold can disagree. Policy priority, override duration, final action, and rollback therefore need to remain visible.

One month of production traffic from nyxr.app

The preceding mechanisms define what the system should keep separate: representation, identity, signal, decision, and learning. Production data can now test whether those boundaries remain visible in the events without conflating several projections of the same request.

Reading outcomes without double counting

I extracted these aggregates from my application nyxr.app. They cover one month of activity, from June 21 to July 22, 2026.

The relational security_events table retains two projections, meaning two representations of the same traffic built for different uses:

• 145,996 requests logged by the gateway with their final outcome;
• 20,878 ModSecurity audit transactions carrying the primary rule match and WAF context.

This scope fixes the denominators and prevents the gateway log from being added to the WAF audit. Once that distinction is established, the data-plane projection can measure the actions actually returned to clients.

Gateway outcomes.

During this window, NyxR logged 8,638 distinct real addresses across 23 services. An address is not an actor and a service is not a homogeneous population. These cardinalities describe the surface; they do not attribute attacks.

Explicit reasons show several active controls: 4,212 CTI blocks across 762 addresses, 3,952 geographic blocks across 1,716 addresses, and 97 blocks from a honeypot, a decoy service designed to attract and report suspicious interactions, across 74 addresses. However, 8,994 records marked blocked lack a specific block_reason, mainly in data predating complete instrumentation. That gap makes the field unsuitable as an exhaustive truth source across the whole window.

Final actions state what was enforced, not which intelligence source contributed to a CTI block. Provenance connects each indicator to the lists carrying it so their measured coverage, freshness, and overlap can be compared.

Which CTI lists actually triggered?

CTI provenance associates the indicator that blocked a request with the public lists that contained it. This separates catalogue size from measured effectiveness. Of the 4,212 CTI-blocked requests, 3,968 remain attributable to the current provenance snapshot; 244 can no longer be attached to it, notably because they predate systematic recording of the exact matched key or concern an indicator that has since left a rotating list.

These counts are not exclusive: a request is attributed to every list carrying its matched indicator. The three rows therefore produce 5,488 attributions for 4,212 unique blocks. That overlap is useful evidence. Data-Shield provides the broadest coverage and dominates stopped traffic; Blocklist.de favors a very fresh window; IPsum adds a cross-list consensus condition. Operationally, provenance should be retained per indicator, each source should expire on its own cadence, and false positives should be measured per list. Adding the counters or judging a source by size alone would produce a false conclusion.

CTI explains decisions based on already-known indicators; it does not describe sequences recognized after several events. Behavioral-ban evidence shows which temporal or application signals actually contributed to that other decision path.

Which behavioral signals contributed most often?

The following table covers evidence stored in the 482 durable bans, not private deployment thresholds or every raw request. One ban can contain several signals; “bans containing signal” counts a ban once, while “evidence occurrences” preserves repetitions in its evidence record.

In total, 387 of 482 bans, or 80.3%, contain at least one of these three families. Only two combine two of the families inside the same evidence record. The dominant fact is therefore not confirmed application exploitation: it is automated reconnaissance centered on paths, file inclusion, and protocol deviations. The strongest defensive gain is to stop those sequences locally, then retain WAF matches and temporal context so the ban remains explainable.

Ban signals describe accumulation over time, while the ModSecurity audit retains the rule engine’s transaction-level view. Classifying its primary identifier compares detected families, but does not turn every match into an effective block.

Recorded rule families.

The table classifies each ModSecurity transaction by its primary rule_id, the numeric identifier of the rule retained as the main cause. It does not recount every secondary rule stored in the audit array.

The contrast between 1,558 SQLi transactions and only 38 WAF actions recorded as blocked demonstrates why “match” does not automatically mean “request denied.” Some transactions run in detection, others may remain below threshold, and policies can differ. Correct analysis requires mode and configuration version.

These aggregates reveal dominant families but erase request order and cadence. A campaign window restores that temporal dimension, then makes it possible to compare computed identity and decision cost with the behavior actually observed.

Campaign, fingerprints, and latency

Real case: a URI-fuzzing campaign.

This URI-fuzzing campaign, automated generation of many paths to discover unexpected routes, files, or behaviors, produced 3,119 requests logged by the gateway in 300 seconds, or 10.4 requests per second on average. It was not a volumetric Distributed Denial of Service (DDoS) attack, which distributes enough traffic to exhaust a resource, but rapid, broad application-path exploration with a scanner signature and 2,100 distinct URIs.

The 3,117 local denials have p50 and p95 of 0 ms in the NGINX log. The maximum in the window is 477 ms. This is not a claim of “zero WAF latency” because timer resolution and log-path semantics constrain the measurement. The important point is that the decision did not wait for the origin or a remote model.

The campaign also demonstrates double-counting risk: 3,121 rule 913100 audit records and 3,119 requests logged by the gateway are nearly twin projections, not 6,240 requests.

The sequence and scanner signature characterize this campaign, but they do not establish a durable client identity. Fingerprints can examine technical continuity across several addresses, provided their limits and experimental role remain explicit.

Measured fingerprints and the JA4 limit.

65,446 logged requests contain a non-empty composite fingerprint, or 44.83% of gateway access records. They cover 935 fingerprints and 6,104 logged addresses. Coherence is 0 on 9,047 records and 1 on 56,399.

224 fingerprints appear behind at least five addresses. Of these, 67 have a non-allowed action ratio at or above 50%. The most distributed fingerprint appears behind 1,235 addresses. None of this proves a botnet: a common stack, CDN, or popular browser can create high dispersion.

The learner, the component that groups recurring observations into proposed profiles, created 24 profiles classified as distributed_attack and 12 active campaigns representing 1,071 requests and 277 cumulative IP memberships. These counts are clustering hypotheses, algorithmic groups of similar observations, not attacker identities.

Finally, 2,218 logged requests have block_reason=fingerprint while their final action is allowed. The fingerprint was computed and logged as an experimental signal, but it did not participate in blocking those requests. Keeping the fields separate prevents a dashboard from turning an experiment into imaginary active protection.

Fingerprint coverage shows which signals were available, not the cost of evaluating them or the other controls. Percentiles by action therefore separate local responses from time spent waiting for the origin before any conclusion is drawn about WAF performance.

Cost and latency.

Percentiles calculated on requests logged by the gateway separate actions. p99 is the value below which 99% of measurements fall, exposing a latency tail hidden by the median.

Allowed latency includes origin time; it does not measure WAF cost alone. Blocked latency mostly measures local response generation. A benchmark is a comparative measurement under controlled conditions: here it should compare the same endpoint, payload, and origin with the WAF off, in DetectionOnly, and On, while separating p50, p95, p99, processor time, memory allocations, body size, and evaluated rule count.

What the WAF can and cannot guarantee

The preceding metrics measure observable decisions and signals; they do not extend the WAF’s knowledge of application internals. Defining that boundary prevents a syntactic or behavioral detection from being presented as proof of exploitation or correct authorization.

A WAF can recognize path-traversal syntax without knowing whether the target file exists. It can identify SQLi-shaped input without seeing the eventual SQL query. It can rate-limit a purchase flow without knowing whether one operator legitimately controls one hundred accounts. It can observe an object identifier without knowing whether the authenticated principal may read that object.

An Application Programming Interface (API) exposes application operations and data to other software through a contract. The OWASP API Security Top 10 2023 places object and function authorization defects among the major risks. A generic WAF does not own the application’s authorization graph: the application must still validate the authenticated identity, object, action, and context.

The WAF is a powerful compensating control, a layer that reduces a risk without fixing its cause in the application, and an observation point. It is not a substitute for secure code.

Evaluating a WAF as a red teamer

The preceding ownership boundaries determine the test method: assessment must compare the representations seen by the edge, WAF, and origin, then attribute each outcome to the component that actually owns the control.

A red team simulates an attacker within an authorized scope to evaluate controls and detection. Its assessment asks more than “which payload gets through?”: it determines which representation each component saw, which decision was applied, and which evidence survived the attempt.

Reproducible evaluation protocol

1. Map the real chain.

Record:

• DNS resolution and any CDN;
• client-side and origin-side protocols;
• HTTP/2 or HTTP/3 downgrade to HTTP/1.1;
• forwarding fields and trusted CIDRs;
• direct origin reachability;
• engine, CRS, and policy versions per virtual host (vhost), the configuration that maps a hostname to one service.

A different response on the origin IP and public hostname is already an architectural finding.

Mapping identifies interpreters and trust boundaries, but it does not prove that they assign the same meaning to the same bytes. Differential tests must therefore vary one representation dimension at a time and compare the output of every layer.

2. Test parser differentials.

Vary one axis at a time:

• duplicate Content-Length, the header declaring body size, interaction with Transfer-Encoding, which describes its transfer coding, and non-canonical whitespace;
• HTTP/2 to HTTP/1.1 conversion, HTTP/2 pseudo-headers prefixed with :, and name normalization;
• duplicate parameters, implicit arrays, and repeated encoding;
• application/json, application/x-www-form-urlencoded, which encodes a form as key-value pairs, multipart, and XML;
• depth, argument count, files, and partial bodies;
• absolute-form URLs, dot segments, encoded slashes, and framework-specific separators.

The expected result is not merely an alert. Compare what the WAF records with what an instrumented lab application receives. A differential fuzzer automatically generates input variants and looks for cases that two parsers interpret differently; it teaches more than a long static payload list.

A differential establishes which representations diverge, but not whether a rule match actually changed the request outcome. Evidence must next separate parsing, detection, scoring, enforcement, and what the origin received.

3. Separate detection, score, and action.

For each test, capture:

This evidence chain connects content to action; it remains invalid if counters and reputation use an identity controlled by the attacker. Address reconstruction, fingerprints, and collisions between legitimate clients must therefore be tested separately.

4. Test identity as a security boundary.

Send forwarding fields from an untrusted source, traverse every legitimate proxy, compare IPv4 with IPv6, and verify the actual key used for CTI, rate limiting, and bans. Rotate IPs behind a stable fingerprint, then place several legitimate clients behind one stable IP.

A valid result can be “the system declines to attribute.” Explicit uncertainty is better than false precision.

Identity tests cover the relationship between clients and decision keys, but automation can distribute activity below every instantaneous threshold. The next step extends analysis to time windows, state cardinality, and distributed sequences.

5. Test time and distribution.

Signatures see one transaction. Modern abuse spreads across:

• low rate over many IPs;
• User-Agent rotation with a stable route sequence;
• expensive endpoints called below the global threshold;
• distributed challenge failures;
• rare sequences whose individual steps look benign.

Measure windows, half-life, key cardinality, Redis memory, NAT collisions, and recovery after expiry.

A control can remain accurate under nominal load yet silently change policy when a dependency fails. Failure tests therefore verify which mechanisms remain local, which move to observation, and which degraded mode actually becomes active.

6. Test failure modes.

In an authorized controlled environment, disable Redis, the scorer, learner, verdict service, PostgreSQL, and control plane. For every failure, document what continues, what moves to observation, what allows by default, what denies by default, and what the operator sees.

An analytical failure must not change parsing. A Redis outage must not silently disable every limit. An invalid bundle must never become active. A full event queue must not block HTTP workers, the processes that execute request handling.

Failure modes establish resilience, but not detection specificity. Each malicious case must therefore be replayed with a benign neighbor of the same shape to determine which property actually triggers the result.

7. Replay with negative controls.

Every malicious case needs a benign neighbor with the same shape: the same Content-Type, the header declaring the body format, size, parameter count, and endpoint. This neighbor is a negative control: it lacks the malicious property and verifies that this property, rather than the general shape, triggers the result.

These controlled cases become comparable only when the lab retains the configuration, inputs, and outputs of every run. A reproducible matrix then turns a one-off bypass into a verifiable and falsifiable experiment.

Building a reproducible lab.

The following harness illustrates a minimal matrix without providing an exploit payload. This shell script uses curl, a command-line client for network protocols, to submit equivalent JSON documents to a lab endpoint, compare outcomes, and preserve response headers.

set -euo pipefail

target="https://lab.example.test/api/parse"
artifact_dir="./artifacts/waf-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$artifact_dir"

for case_name in baseline duplicate-key deep-json oversized-fields; do
  curl --http2 --silent --show-error \
    --dump-header "$artifact_dir/$case_name.headers" \
    --output "$artifact_dir/$case_name.body" \
    --header 'Content-Type: application/json' \
    --data-binary "@$case_name.json" \
    --write-out '%{http_code} %{time_total}\n' \
    "$target" | tee "$artifact_dir/$case_name.result"
done

sha256sum "$artifact_dir"/* > "$artifact_dir/SHA256SUMS"

sha256sum computes a SHA-256 fingerprint for every artifact; any later modification changes that value and becomes detectable. The same run should capture the bundle version, WAF mode, PL, threshold, full audit record, gateway log, normalized request seen by the origin, and resource cost. The experiment then becomes reproducible and falsifiable.

The lab produces detailed evidence for every layer. Those observations can then be condensed into architectural invariants that remain valid when the payload, address, or tool signature changes.

Ten invariants that survive evasion.

1. Parse before classifying. Content the WAF cannot parse needs an explicit policy.
2. Reduce interpreters. Normalize early, avoid unnecessary downgrades, and align edge with origin.
3. Establish identity before counters. A rate-limit key derived from a spoofable IP is not a control.
4. Keep decisions local. The hot path must not wait for a database or remote model.
5. Separate signal, reason, and action. A decision computed only for observation must not appear as an effective block.
6. Version policy. An alert without a configuration version is hard to reproduce.
7. Prefer narrow exclusions. Remove the false positive, not the protection family.
8. Bound every state. Body, depth, arguments, cardinalities, queues, history, and Time To Live (TTL), the duration after which state expires.
9. Measure false negatives and false positives. A false negative is an undetected attack; block rate alone rewards unusable systems.
10. Treat the WAF as one layer. Authorization and business invariants remain in the application.

The mental model to retain

The preceding protocol turns isolated tests into durable properties. Taken together, its invariants lead to a model of the WAF as a distributed system that sees only a partial representation of the request and its sender.

The modern WAF is a partially observable distributed system. The data plane receives a representation of the request, not operator intent. It establishes an HTTP boundary, reconstructs a probabilistic identity, applies local policy, and emits evidence. Other components correlate that evidence over time, but they cannot retroactively rewrite what happened.

Four questions resolve many evasion claims:

1. Which exact representation did the WAF parse?
2. Which representation did the origin finally consume?
3. Which signal contributed to which decision under which policy version?
4. Which independent layers remain active if that signal disappears?

The production results I present from nyxr.app make this model concrete. NyxR absorbed a URI-fuzzing campaign locally, computed and logged a fingerprint without using it to block, maintained the gateway when the verdict service timed out, and distinguished a ModSecurity detection from an effective block. These facts make the system testable, which is more useful than a promise of “intelligent protection.”

References

• IETF, RFC 9110: HTTP Semantics, 2022.
• IETF, RFC 9112: HTTP/1.1, 2022.
• IETF, RFC 9113: HTTP/2, 2022.
• IETF, RFC 9114: HTTP/3, 2022.
• OWASP ModSecurity, Reference Manual v3.x.
• OWASP Core Rule Set, Anomaly Scoring, Paranoia Levels, and Rule Exclusions.
• OWASP Core Rule Set, published CRS releases.
• NGINX, ngx_http_realip_module.
• OpenResty, lua-nginx-module, Lua phases and shared dictionaries.
• OWASP, API Security Top 10 2023.
• Kettle, HTTP Desync Attacks: Request Smuggling Reborn, Black Hat USA.
• Kallus et al., The HTTP Garden: Discovering Parsing Vulnerabilities in HTTP/1.1 Implementations by Differential Fuzzing of Request Streams, 2024.
• Akhavani et al., WAFFLED: Exploiting Parsing Discrepancies to Bypass Web Application Firewalls, 2025.
• Demetrio et al., WAF-A-MoLE: Evading Web Application Firewalls through Adversarial Machine Learning, 2020.
• FoxIO, JA4+ Network Fingerprinting.