proxyfile

Conformance

The conformance battery runs a proxy on the configuration proxyfile renders for it and checks that the proxy answers requests as the proxyfile says. It is the Go package conformance, whose files are test files only, so the released program never carries them. It runs each target of this version: Envoy on Envoy 1.40.2, nginx on nginx 1.30.6 and 1.31.8, each on its plain image and on its image tagged -otel, and HAProxy on HAProxy 3.2.26 and 3.4.7.

What the battery proves

Each entry of the battery is a proxy image and the proxyfiles it runs. Envoy's entry, envoy-1.40, runs fourteen: examples/full/hocon/Proxyfile, which writes every key but the few its opening comment names, examples/portable/hocon/Proxyfile, which writes in the same world only the keys every generator is to render alike, nginx's own eight files, conformance/testdata/nginx.yaml and conformance/testdata/nginx-plain.yaml, the same with no pattern route, and, with an auth check, conformance/testdata/nginx-checks.yaml and conformance/testdata/nginx-auth.yaml, and, with tracing, conformance/testdata/nginx-tracing.yaml and conformance/testdata/nginx-tracing-by-name.yaml, its collector found by name, and, with a connection limit, conformance/testdata/nginx-limits.yaml, and, with client_address: connection and no tls, conformance/testdata/nginx-connection.yaml, and HAProxy's own four, conformance/testdata/haproxy.yaml, conformance/testdata/two-checks.yaml, conformance/testdata/haproxy-traffic.yaml and conformance/testdata/haproxy-limits.yaml. The nginx entries nginx-1.30 and nginx-1.31 run on the two images pinned in scripts/nginx-images, one entry for each line, and run nginx's own files but the two with tracing, examples/full, which nginx refuses, and examples/portable. The nginx entries nginx-otel-1.30 and nginx-otel-1.31 run on the two images pinned in scripts/nginx-otel-images, the same nginx versions in the images tagged -otel, which carry nginx's OpenTelemetry module, one entry for each line, and run the two files with tracing alone: a file with tracing loads that module, which the plain images do not carry (nginx). Each line of scripts/nginx-otel-images names an image tagged -otel and each line of the other files names one that is not, pinned by digest, as nginx:<major>.<minor>.<patch>-otel@sha256:<digest> or the same without -otel; the battery stops before any container at a line that does not, and scripts/pinned-images.sh takes no image from such a file. The HAProxy entries, haproxy-3.2 and haproxy-3.4, run on the two images pinned in scripts/haproxy-images, one entry for each line, and run HAProxy's own four files, examples/full, which HAProxy refuses, and examples/portable, on which they run no row, as its rows assert the Envoy output's answers to keys HAProxy writes with its own features. Each file is rendered for the entry's target with proxyfile render --target <name> through the program's own command line, the path a user takes. The proxy then runs on the rendered file in its own image, among stub backends, and the same requests get the answers the format promises.

conformance/testdata/<proxy>.yaml is a proxy's own file, and <proxy>-<name>.yaml another of its own: the world of examples/full, or of examples/portable for HAProxy, with only the keys that proxy's generator writes in this version, so the rows whose keys it writes run on that proxy before its generator writes every key. Envoy's entry runs it too, so its answers are Envoy's, measured as every other answer is. nginx's nginx.yaml writes tls with clients, client_address: connection, remove_from_clients, cors and the lists of every environment as examples/full does, but remove_from_clients names only x-client-id, because the generator refuses a name with a * prefix. It also writes limits: {request_headers: {size: 60KiB, count: 100}}, Envoy's own defaults; timeouts: {idle: 2s, upstream_idle: 3s}, an idle time short enough that a row sees a silent service ended without waiting long; and log: {status_from: 400, sample: 1%}, examples/full's, so the battery runs every form of those keys the generator writes. Its traffic keys are examples/full's: catalog found by its DNS name through dns: {resolver: "10.0.0.2:53"} and balanced by the hash of x-session-id, the catalog route limited to 20 requests a second with a burst of 10 by client, the rpc entry's retries: 1 and its limit of 100 a second for all, and shop balanced by least_requests; but the shop route writes retries: 0, which tries each of its requests once, so the rows that send it a body meet the form nginx streams, and two routes of their own to shop write retries: 1, /retried/ and /retried-limited/, the second limited to 10 requests a minute for all, which the rows that need a retry send to; and a service sessions at shop's two addresses, balanced by the hash of x-session-id, takes the route /sessions/, which no limit holds, so the hash rows send to a route the rate limit rows do not spend. nginx-plain.yaml writes no tls, keeps client_address at forwarded, the default with no tls, and removes x-request-id from clients, which nginx reads to keep a client's own request id under forwarded. It writes none of limits, timeouts and log, so it runs each proxy's defaults, and it names the service hold, at 10.0.2.30 on port 8095, behind the route /hold/, and the service held, at 10.0.2.30 and 10.0.2.31 on port 8095, to which its rpc entry sends example.hold.* and its route /held/ its requests, each retrying once (The world). It retries no route of /retried/, so its shop route serves the rows of a retried request that send there and tries each request once. Its route /limited/ admits one request a minute by client, and only the client keys' rows send to it, so each finds the bucket of the client it names as the rows before it left it. HAProxy's two-checks.yaml is nginx-plain.yaml with an auth check and without the hold and held services, the rpc entry's retries and the routes /held/ and /limited/: the check service, checks, has two addresses, the battery's check-1 and check-2, so a row can make one refuse connections; its when_unavailable is deny, where haproxy.yaml's is allow, so each answer the check cannot give runs in both modes; it asks the check only of a request with an x-api-key header, which HAProxy's script passes on line by line, where haproxy.yaml's authorization is one it joins into one line, so a row that sends neither is served as on nginx-plain.yaml; it accepts x-user-check alone, in which the stub's check reports what it was told; and remove_from_clients also names the prefix ho*, which covers Host.

HAProxy's haproxy-traffic.yaml writes the keys HAProxy writes with its own directives: header limits of 96 KiB and 150 lines, an idle time of 2 s, retries on the shop route, an rpc entry without retries, a rate limit of 30 requests a second by client on the catalog route, a service sessions hashed by x-session-id and a service checked with health checks every second, both at shop's two addresses, and a log of every answer of 400 or more. It runs the rows of the scenarios timeout, retry, rate limit, hash and traffic keys alone. A key a target writes with a feature of its own is held to its main behaviour, not to every answer the Envoy output gives with it: the target names the rows that run that behaviour, the key's main rows (target.mainRows), and runs no other row that holds the key, on any of its files, and the coverage checks ask it for no other. HAProxy names main rows for the keys of haproxy-traffic.yaml, and nginx for health, its passive check, whose rows are those of the scenario connection limit and failed backends. Envoy's entry runs the file whole, so the answers of its rows are Envoy's.

nginx's nginx-limits.yaml writes limits: {connections: 2} and the shop service at its two addresses behind one route, with no tls, and runs the rows of the scenario connection limit and failed backends alone. It writes health: {path: /healthz, unhealthy: 1} on shop, so on nginx one failed request takes a backend out for 10 s. Envoy's entry and each nginx entry on the plain image run it. HAProxy's haproxy-limits.yaml writes the same limit with the shop service at one address, so HAProxy's own maxconn on that server binds, and runs the row of the connection limit alone; Envoy's entry and each HAProxy entry run it.

nginx's nginx-connection.yaml writes client_address: connection with no tls, so its clients speak plain HTTP, the auth check with send_request: true and one route to shop, and runs the rows of the scenario the scheme the check is told alone, so they show what each proxy tells the check over plain HTTP under connection: an absolute https target, which the Envoy output answers 403 and nginx serves, telling the check the connection's scheme, an HTTP/2 request with :scheme https, which the Envoy output tells the check as https, and a client's own x-forwarded-proto. Envoy's entry and each nginx entry on the plain image run it.

nginx's nginx-checks.yaml is two-checks.yaml for nginx: the same check service with two addresses, when_unavailable: deny, the x-api-key condition and send_request: true, with remove_from_clients naming x-request-id alone, because the nginx generator refuses a name with a * prefix there, so each answer the check cannot give runs on nginx under deny, and a route /retried/ to shop that writes retries: 1 and is checked as its shop route is. nginx-auth.yaml is nginx.yaml with an auth check under allow: only_with_header: authorization, send naming authorization, x-country, x-client-* and http2-settings, accept naming x-user-id, x-tier-level and x-user-check in full, and send_request: true. It writes auth: off on accounts-events, accounts-webhooks and a public prefix route, an action on two rpc methods and reports on one, beside nginx.yaml's tls with clients, client_address: connection, pattern routes, cors and log, so the rows of the check run beside every form of the keys the generator writes. Of the traffic keys it writes only the catalog route's limit, nginx.yaml's, so the rate limit rows that need the check run on an nginx file, and retries: 1 on a route /retried/ to shop, checked as its shop route is, so the rows of a retried request that the check allows, denies or cannot answer run under allow. Envoy's entry runs both files too, so each literal of their rows is measured on Envoy. nginx-tracing.yaml is nginx-auth.yaml with no tls and no client_address, so its clients speak plain HTTP and forwarded applies, and a row's own x-request-id reaches the draw that picks the requests traced, and with tracing: the otel provider sending its spans to the battery's collector at 10.0.3.40:4317 (The world), sample: {rate: 5%}, path_in_url: true and eleven tags, of each source the nginx generator reads a tag from: an environment variable with a default, unset, and one set, x-request-id, a header with a default and one without, a header whose lines Envoy joins, the check's first accept name, a default holding $, " and \, a tag named like a field of the Envoy output's span, component, and two of variables named like a property every object has, constructor, unset, and toString, set. Envoy's entry runs it too, so each literal of its rows is measured on Envoy. nginx-tracing-by-name.yaml is the same tracing with the collector named collector.internal.example:4317 and dns.resolver naming the battery's resolver, which gives that name the collector's address, so nginx sends its spans through the relay it writes for a name (nginx), and with only the shop service and route. It runs the rows of the tracing scenario alone, the ones its keys take; nginx-tracing.yaml runs every other row. Envoy's entry and each nginx entry on the image tagged -otel run it.

A file that writes no tls is sent over plain HTTP: HTTP/1.1, and HTTP/2 with prior knowledge. A file with tls is sent over TLS, with a client certificate the run's clients CA signed (The world). Every request goes to the proxy's one listener, and the client asks for the host the request names as its TLS server name, without the port, as a browser does, and checks the server certificate against localhost, the name it carries; an empty host, a host that is an IP address, and one that holds a character other than a letter, a digit, a dot or a hyphen ask for localhost. A row may name a server name of its own. nginx answers 421 to a request whose Host is another name than its TLS server name when it asks for client certificates, so a client that asked for one name whatever the host a row sends would read that refusal in every row that names another host.

Each row names the core keys its answer depends on and runs on each file that writes them all, counting the keys of the origins file a proxyfile names and of the methods its rpc.proto reads; a row that names none runs on every file. Before any container starts, the battery fails when a row names a key that is not a core key of the format, when a row runs on no file of Envoy's entry, where its answers were measured, when a row holds an answer for a file label no entry carries or for a file the row does not run on, when a row that runs on a file of Envoy's runs on none of the files of a proxy version's entries while that target refuses none of the row's keys, when a key Envoy's rows exercise is exercised on none of the files of a version's entries while that target does not refuse it, and when a count of rows this page gives below is not the number of rows the entries run. The entries of one version are judged together, nginx 1.31's on the plain image and on the image tagged -otel among them, and the entries of two versions never are. A row checks only what every proxy must answer: the status, which service and which of its backends answered and what it received (port, protocol, method, path, host and headers), the response headers that matter, grpc-status and a time window. On one file a row's answer is the same for every target unless the target declares a difference that changes it (below); where two files give different answers, as the rate limit's count and a route only one file writes do, the row holds each file's answer. Every answer was measured on Envoy 1.40.2, and conformance/scenarios_test.go notes beside each scenario what Envoy also answers that no row checks. A scenario's rows run one request at a time, but for the cost scenario's flood and fifteen rows that wait for a proxy's timer or its close: a service that takes no connection within 5 s, a retried request and a retried gRPC call whose every try times out connecting, the six rows over HTTP/2 that wait for Envoy to close the connection after it refuses the request, and the six timeout rows; and 54 rows of the tracing scenario. Each of those sends its request as its scenario starts, on connections of its own, and is checked once the scenario's other rows are done; the next scenario starts only after the last of them ends. A row that reads spans, one of those 54 or a row in sequence, is judged once the scenario's rows in sequence are done, on the spans the collector held 6 s after the last request of the scenario's rows that read spans, and on those it holds 6 s later when its span is still missing or it wants none: nginx's exporter and the Envoy output's tracer each send the spans they hold every 5 s. On full, portable, nginx-file and nginx-auth, which limit the catalog route to 20 requests a second, one row of the path normalisation scenario waits one second before it is sent, so the route's bucket fills again between the 10 of the scenario's rows that route serves before it and the 11 from it on, however fast a runner sends them; routing's and CORS's rows send it 9 more before them, so 19 reach the route before that wait on portable, whose bucket holds 20. The scenarios run in this order: in Envoy's entry 610 rows on full, 518 on portable, 349 on nginx-file, 259 on nginx-plain, 381 on nginx-checks, 531 on nginx-auth, 576 on nginx-tracing, 39 on nginx-tracing-by-name, 3 on nginx-limits and 6 on nginx-connection, nginx's own files, and 471 on haproxy-file, 364 on two-checks, 37 on haproxy-traffic and 1 on haproxy-limits, HAProxy's own; in each nginx entry on the plain image 349 on nginx-file, 259 on nginx-plain, 381 on nginx-checks, 531 on nginx-auth, 518 on portable, 3 on nginx-limits and 6 on nginx-connection; in each nginx entry on the image tagged -otel 576 on nginx-tracing and 39 on nginx-tracing-by-name; and in each HAProxy entry 471 on haproxy-file, 364 on two-checks, 12 on haproxy-traffic and 1 on haproxy-limits. That is 4,145 rows in Envoy's entry, 2,047 in each nginx entry on the plain image, 615 in each nginx entry on the image tagged -otel and 848 in each HAProxy entry, 11,165 in all, and each passes. The battery counts these rows before any container starts and fails when one of these numbers is not what it counts. Before nginx wrote rate limits, RPC retries, balance, hash and DNS names the same entries ran 3,041, 1,310 and 791 rows: 40 rows were added, among them the scenarios client keys and hash, nginx's files gained the traffic keys those rows need, and each nginx entry runs examples/portable, where it ran no file of that world. Before nginx wrote HTTP retries they ran 3,273, 1,937 and 823: 13 rows were added, those of a retried route in the scenarios retry, auth and rate limit, nginx.yaml, nginx-auth.yaml and nginx-checks.yaml gained the routes those rows send to, and so the three rows of routes[].retries the battery had run on full and nginx-file alone run on nginx-auth and nginx-checks too. With those they ran 3,326, 1,977 and 823; since then 3 rows were added in the scenario retry, a POST with neither Content-Length nor Transfer-Encoding and a GET in one-byte chunks, each first connection refused, and a retried request whose every try times out connecting, and nginx-plain.yaml gained its retried route /held/, so the rows of routes[].retries run on it too. Before nginx wrote tracing they ran 3,353, 2,001 and 823: nginx-tracing.yaml and its two entries were added, where each nginx entry on the image tagged -otel and Envoy's entry run the 503 rows of the other scenarios that the file's keys take, and 42 rows were added in the scenarios tracing and tracing a retried request, which run on full and nginx-tracing. Before nginx wrote a collector found by name they ran 3,940, 2,001, 545 and 823: nginx-tracing-by-name.yaml was added, on which Envoy's entry and each nginx entry on the image tagged -otel run the 25 rows of the scenario tracing that its keys take. Before nginx's tracing wrote the span only for a target sent as written, read the client's trace as Envoy reads it and sent the client's bytes of it on, they ran 3,965, 2,001, 570 and 823: one row was added in the scenario routing, an absolute target of another scheme than http and https, which runs on every file, and 20 in the scenario tracing, of which 14 run on nginx-tracing-by-name. Before nginx wrote the connection limit and the passive health check they ran 4,028, 2,006, 605 and 825: nginx-limits.yaml was added, and three rows in the scenario connection limit and failed backends, one of limits.connections and two of health, each on full and nginx-limits. Before HAProxy wrote its remaining keys with its own directives they ran 4,034, 2,009, 605 and 825: haproxy-traffic.yaml was added, seven rows were added, six in the scenario traffic keys and one in the scenario hash, haproxy-limits.yaml was added for the row of the connection limit, and haproxy.yaml gained a service found by DNS, so the row of dns.resolver runs on haproxy-file. Before nginx told the auth check the connection's scheme under client_address: connection and the battery ran each proxy with two variables set they ran 4,102, 2,026, 610 and 839: nginx-connection.yaml was added with the scenario the scheme the check is told, whose rows run on full, portable, nginx-auth, haproxy-file and nginx-connection, and those whose keys hold no client_address on two-checks, nginx-checks and nginx-tracing too, and two rows were added in the scenario tracing, each of an env tag whose variable is set, one of them named like a property every object has, which run on full and nginx-tracing. In order:

Scenario Keys What its rows check
routing routes[].pattern, routes[].path, routes[].method, routes[].rewrite, routes[].prefix, routes[].to, services.<service name>.http, services.<service name>.addresses, services.<service name>.address, services.<service name>.host_header, dns.resolver, origins.<environment name>.hosts, client_address, tls.certificate, limits.request_headers.count, limits.request_headers.size A pattern route, and a path its pattern misses going to the prefix route; a path and method route with its rewrite, and another method going to the prefix route; a service whose address the resolver gives; the prefix route over HTTP/1.1 and over HTTP/2, where the service still hears HTTP/1.1; the host each service hears, host_header's value for accounts; a host of another environment; 404 for a host no environment names, with no content-type, with vary: Origin and with an x-request-id the proxy makes; a service at its one address; a path compared by letter case, /API/webhooks going to the prefix route; the path /_pf_auth, written and with its underscore escaped, routed as any other; an escaped newline, %0A, at the end of a path and inside one, matched by the patterns as written and passed on as sent; an upgrade other than WebSocket answered 403, and an h2c upgrade served as a plain request whose service receives no upgrade or http2-settings, which HAProxy answers 400, a declared difference; the scheme the request came in on, in x-forwarded-proto; with client_address = connection, a client's own x-forwarded-proto replaced by the connection's scheme, and its own x-envoy-external-address and x-request-id by values the proxy writes; each of 32 requests that carry no x-request-id gets one the proxy makes, a version 4 UUID in lower case, 8-4-4-4-12 hexadecimal digits with the third group starting with 4 and the fourth with 8, 9, a or b, and the answer carries it too; 502 with no content-type for a service that refuses connections, and 502 with no body for a service that takes no connection within 5 s ("a service that takes no connection within 5 s is answered 502"), which only nginx-plain has, as its hold service behind the route /hold/ (The world): Envoy and nginx answer it after at least 4.95 s and under 5.95 s, the connect time of both with no timeouts, and every other file serves /hold/x from shop; a service's X-Accel-Redirect passed on to the client, not followed; a service's Keep-Alive not passed on to the client; the headers the Envoy output removes (proxy-connection, x-forwarded-client-cert and the x-envoy- headers that steer an Envoy) never reach the service; the host of an absolute request line, not the Host header, is the host the service hears, where HAProxy answers every absolute target 400, a declared difference; an Upgrade that Connection does not name is removed and the request served; header names holding _ or . reach the service; TE: gzip, trailers reaches it as trailers; 404 for a host with a dot at its end, with no content-type, as the 404 of an empty X-Forwarded-Proto and the 403 of an upgrade other than WebSocket from no origin; 400 for a method Envoy's parser does not know, with no x-request-id, and after the 400 of a byte above 0x7F in the target, of a header name holding @ and of HTTP/1.10 the connection closed, so a request written after it gets no answer, and after the 426 of HTTP/1.9 and the 403 of an upgrade other than WebSocket, from no origin or a foreign one, closed too, where HAProxy keeps it and serves the request after the body, a declared difference; a QUERY with a Content-Type served and one with none answered 400, over HTTP/1.1 and HTTP/2, the 400 with no content-type and no x-request-id, before the 404 of a host no environment names too; a QUERY whose Content-Type is spaces alone answered 400 over HTTP/1.1 and served over HTTP/2, where HAProxy answers it 400 over HTTP/2, a declared difference, and one whose Content-Type is a tab answered 400 over HTTP/1.1; a pattern's {,3} read as text, as RE2 reads it; a long path `((a
mutual TLS tls.certificate, tls.key, tls.clients.ca, origins.<environment name>.hosts A client with no certificate, one whose certificate another CA signed, and one whose certificate the clients CA signed and whose validity ended reach no service: the handshake fails, or the proxy refuses it, where nginx completes the handshake and answers an empty 400, the difference nginx's row of tls.clients.ca names. A request whose TLS server name is one host of an environment and whose Host is another host of the same environment is served by Envoy and answered with an empty 421 by nginx, which asks for client certificates, a declared difference. The other CA carries the clients CA's name, so a proxy that matched the issuer by name would let it through. Every other row sends a client certificate the clients CA signed. A client that offers only TLS 1.1, a TLS 1.2 client that offers only a CBC suite, and one that offers only the group P-384 are refused in the handshake, and one that offers only the group X25519MLKEM768 is served. Eight rows, on every file but nginx-plain, two-checks and nginx-checks, which write no tls.
CORS cors.allow_methods, cors.allow_headers, cors.expose_headers, cors.max_age, origins.<environment name>.hosts, origins.<environment name>.origins, origins.<environment name>.patterns A preflight from a listed origin and one from an origin a pattern covers answer 200 with access-control-allow-origin and access-control-max-age; a preflight and a request from another origin answer 403, and so does a request from another origin to a path ending .html, as text/plain on every proxy, whose own answers take no type from the path's ending; a request from a listed origin reaches the service with its origin and is answered with access-control-allow-origin and access-control-expose-headers. The preflight from a listed origin and the request from another origin carry a token the check denies and one the token filter refuses, each answered 401 alone, so on a file that writes the check their 200 and 403 show that the origin is decided before the token and the check. An origin on the environment's own host that neither list holds, https://api.example.org to api.example.org, is admitted without access-control-allow-origin: its request reaches the service and its preflight answers 200. With no list, an origin on one of the environment's hosts is admitted and every other refused: a request with no Origin, and one from http://localhost over http or https, with a port, or from the environment's other host http://127.0.0.1 reaches the service, and a preflight from http://localhost is answered 200 by the proxy; an origin on another environment's host, the host in capitals, with a trailing dot or with other characters for its dots, null, a present but empty Origin, a preflight from another origin and a WebSocket upgrade from one answer 403, where HAProxy's parser answers that upgrade, which carries no Sec-WebSocket-Key, 400, a declared difference. An empty Origin over HTTP/2 is answered 403 too, and an empty Origin line before or after the origin of the host, or a tab before that origin, changes nothing: the service receives the origin as one line. A preflight from a listed origin with no token is answered with the method and the headers it asks for; one from a listed origin in capitals is admitted with the origin as sent, and one from a pattern's origin in capitals is refused. The Origin header is read as one line: two Origin lines a pattern covers when joined are admitted with the joined line in access-control-allow-origin, where nginx answers 403, a declared difference, two whose first is listed, two of the request's own origin, and one line holding a listed origin after a comma are refused, and an empty Origin line after a listed one is left out; a preflight that asks on two lines for two methods or for headers is answered with both, joined with a comma, and by nginx with a comma and a space, a declared difference. A listed origin with a trailing space is admitted, and the service receives it without the space. A service that is down, asked from a listed origin, is answered 502 with access-control-allow-origin and access-control-expose-headers, and a request from a listed origin to a host no environment names is answered 404 with no access-control-allow-origin. A refused path, a preflight to it, an upgrade other than WebSocket, a request with a NUL byte in a header from a listed origin and one with Content-Length beside Transfer-Encoding: chunked from it are answered with no access-control-allow-origin, where HAProxy serves the last with it, a declared difference, and a header line of 9,000 bytes from one is served with it by every proxy, nginx's header buffers reading a line as long as the header limit (nginx). An admitted Origin of 600 bytes is echoed beside a service's answer of 100 KB, and so is one of 1,100 bytes, where HAProxy answers 403 before the service, a declared difference. A preflight from a listed origin to /..;x/products/1, with no token and with a token the check allows, is answered 200 by Envoy, which resolves the path and asks no check, and 400 by nginx and HAProxy, which with the token ask the check first, each a declared difference, and a denial of the check from a listed origin carries access-control-allow-origin and access-control-expose-headers. Fifty-two rows, all on full, portable, haproxy-file and nginx-auth, fifty on nginx-file, which writes no check, and nineteen on nginx-plain, two-checks and nginx-checks, which write no list of origins and no cors.
auth auth.ask, auth.only_with_header, auth.send, auth.accept, auth.send_request, auth.timeout, auth.when_unavailable, routes[].auth, remove_from_clients, auth.jwt.claims, services.<service name>.host_header, cors.expose_headers, origins.<environment name>.origins, log A request with no authorization header reaches the service, which it could not if the check were asked; when the check allows, the service receives the check's headers auth.accept admits and not another, and what the check was told (the method, the path with its query, x-country), never the x-forwarded-uri and x-forwarded-method the client sent; the check's 401 and 403 reach the client, without the x-envoy-upstream-service-time Envoy's client for the check adds; a check that answers 503 lets the request through, as when_unavailable = allow says, and on two-checks and nginx-checks, whose when_unavailable is deny, gets 403; the mark of a request the check could not answer reaches the service, and a client's own copy never does, whether the check allows or is not asked; a route with auth = off is not checked; a client's own copies of the headers remove_from_clients names, of the check's headers and question, and of the headers auth.jwt.claims sets never reach the service. A header remove_from_clients names does not reach the service when a client sends it on two lines, one with its name in capitals; a client's own x-request-id is replaced by another when remove_from_clients names it; and on two-checks, whose remove_from_clients names the prefix ho*, a client's ho-x does not reach the service and its Host does. The check is told it is internal and its deadline, x-envoy-internal: true and x-envoy-expected-rq-timeout-ms: 1500; a header sent on two lines reaches it as one line, joined with a comma; a client's own x-forwarded-proto reaches it under forwarded, and the connection's scheme under connection; it receives the Host the client sent whatever the check service's host_header; and the only_with_header header sent twice, an empty line before or after the token, asks it. A check's 201 is answered 403, and a status of 600 is passed on as the check's own, where HAProxy and nginx answer 403. A denial of 20,000 bytes, and one with a header of 50,000, reach the client, and an allowing answer whose x-user-id holds 20,000 bytes sets it on the request where auth.accept admits it, where HAProxy answers each 403, a declared difference. An answer whose body breaks off, and one of 101 header lines, are a check that could not answer: the request is served with the mark under allow and answered 403 under deny. A 200 that RFC 9112 does not read whole is such a check for Envoy too, in nine forms: a Content-Length that is not a number, two Content-Length lines, a control byte in a header value, a status that is not three digits, a header name outside the token characters, Content-Length beside Transfer-Encoding, a chunk size that is not hexadecimal, a malformed header line in a 1xx before it, and a malformed trailer line; HAProxy answers each 403 under allow too, a declared difference. Of 24 requests while check-2, one of the two check backends of two-checks and nginx-checks, refuses connections, at least one is answered 403 on those files: no other backend answers a check it refused. Rows for each form of an answer a check can write that RFC 9112 reads in a way of its own hold Envoy's reading, measured on Envoy 1.40.2: a status line of 2 0 0, 20 or 000, of 304 or of 101 is passed on as the check's own, and so is a denial with a folded header line; a status of 200.1, 500, 599, or 503 with a body, a 200 with a header line with no colon, a header value with a vertical tab, Transfer-Encoding: gzip, chunked, or identity beside a Content-Length, a chunk that does not end its line, a space before a header's colon, a header name that holds a NUL or the byte 0xE9, and a Content-Length past what any reader holds, are each a check that could not answer, and so is a denial with a header line with no colon; a 200 of HTTP/2.0, with two spaces after its version, with a space after a chunk's size, with spaces around its Content-Length, with a lone CR before a CRLF, with _ in a header name, with bytes after its body, with a body of 1 MiB, or holding the relay's own header name x-proxyfile-check, is read as allowed. A denial with a body of 1 MiB, of 1 MiB and a byte and of 1,100,000 bytes reaches the client whole, and a denial's header that holds the byte 0xE9 reaches it as the check wrote it, and a denial with no body reaches it with no content-type. A 200 whose chunk-size line ends in a bare LF, or whose last chunk's line does, is a check that could not answer on Envoy, and a 200 whose chunk data, trailer line and final line end in a bare LF is read as allowed; nginx's relay and HAProxy's script refuse the first two under allow too, a declared difference. A 200 whose accepted header is folded reaches the service with the spaces and tabs of the fold, a tab, spaces on both lines or two folds, where HAProxy joins them by one space, a declared difference, and one whose accepted header is empty reaches the service empty. A 200 whose accepted header takes the request past the header size is answered 500, where HAProxy answers 431, a declared difference, and one whose accepted headers take it past the header count, 100 fields as Envoy counts them, is answered 500 by Envoy on every file but full and portable, which serve it, by nginx and by HAProxy, with an admitted origin's CORS answer, where one they take to the count is served: with client_address: connection the x-forwarded-for and x-envoy-external-address the proxy sets count too, so 87 lines from the client reach the count there and 91 elsewhere. A 200 that sets no accepted header is not counted, so with 96 numbered lines beside it the request is served by every proxy. With the accepted header and 91 numbered lines, a request with a chunked body is served, its Transfer-Encoding line not counted, and one with a body framed by its length is answered 500, its Content-Length line counted, on two-checks and nginx-checks, where haproxy-file and nginx-auth, with three accepted headers and the two the proxy sets under client_address: connection, answer both 500 and full and portable, whose count is 150, serve both; and a gRPC call over HTTP/2 whose accepted header and 87 numbered fields take it past the count is answered 200 with grpc-status 2 and reaches no service, and one with 86 is served, where haproxy-file and nginx-auth answer 86 with grpc-status 2 too and full and portable serve both. nginx's relay reads some of these forms otherwise, and each such row holds nginx's own answer under by, named by the row of auth.ask or of auth.when_unavailable that declares it. Rows for how nginx asks the check hold Envoy's answer, and nginx's own where it declares a difference: the check location's own path asks the check as any other path does; an upload sent with Expect: 100-continue that the check refuses gets 100 Continue first; a request with a body the check allows reaches the service with its body; a sent header that holds the byte 0xE9 reaches the check with a ! for it, one that holds well-formed UTF-8 reaches it as sent, one that holds well-formed UTF-8 and then a byte that is not reaches it with a ! for that byte, a header on two lines of characters outside ASCII reaches it as sent, joined by a comma, and a path of well-formed UTF-8 and then a byte that is not, over HTTP/2, is told to it escaped, as the service receives it; an only_with_header header sent as a lone tab is not asked; 100 checked streams at once on one HTTP/2 connection are each answered by their own check; an absolute target's host is the host the check is told, whatever the Host header; a client's own user, question and unchecked mark do not reach the service when the check allows with none of them, and the check receives none of a client's own removed header, user and mark; a dot segment with a ; parameter asks the check first, and is answered 400 when the check allows; a denial of a request from a listed origin carries the CORS answer; a WebSocket upgrade the check allows reaches the service, and one it denies is answered with the denial; a header the check's answer sets on one request is not on the next request of the same connection, and a check's 2xx other than 200 is answered at once on a kept-alive connection; a request the check denies, one whose check is unanswered and one whose check's status is passed on each have a body that holds a request read as a body, and the next request on the connection is answered, where nginx and HAProxy answer a status of 600 403 and the unanswered one is 403 first on two-checks and nginx-checks; a client's own x-proxyfile-method and x-proxyfile-path change nothing the check is told; the check is told the path with an escaped letter, with a dot-dot segment and a query, with a query's escapes of unreserved and of reserved characters, and with a quote; an HTTP/2 request is told its :scheme; a client's HTTP2-Settings of an h2c upgrade does not reach the check, and with no upgrade it reaches it when send covers it; a check's redirect whose Location is a path reaches the client as the check sent it; and the request log holds how long a denied request's check took, no check time for a request with no only_with_header header, and an unanswered check for a request the check cannot answer under allow. A request to a retried route is checked once, before its first try: with one shop backend refusing connections, each of 20 requests the check allows reaches the other backend with the header the check accepted, each of 6 it denies is answered 403 with no service, and each of 10 whose check answers 503 reaches the other backend with the unchecked mark under allow and is answered 403 under deny. One hundred and fifty-two rows: all on full, one hundred and fifty-one on nginx-auth, which writes no auth.jwt.claims, one hundred and forty-eight on portable, which writes neither that nor routes[].retries, one hundred and forty-four on haproxy-file, which writes none of those and no log, one hundred and thirty-two on nginx-checks, which writes none of auth.jwt.claims, auth.send and log, no route with auth = off and no list of origins, one hundred and twenty-nine on two-checks, which writes no routes[].retries either, and the three rows of remove_from_clients also on nginx-file and nginx-plain, which write it and no check.
the scheme the check is told client_address, auth.ask, auth.accept, auth.send_request Each row asks the check, which reports the scheme it was told, and the service reports the x-forwarded-proto it received. Under client_address: connection: an absolute https target is served over TLS and told https, but by HAProxy, which answers an absolute target 400, and on nginx-connection, over plain HTTP, the Envoy output answers it 403 and asks no check, and nginx serves it and tells the check and the service http, the connection's scheme; an HTTP/2 request with :scheme https is told https over TLS, and on nginx-connection the Envoy output tells the check https and sends the service http, where nginx tells both http, a declared difference; and a client's own x-forwarded-proto of https or ftp is replaced by the connection's scheme for the check and the service, by every proxy. Under forwarded, on two-checks, nginx-checks and nginx-tracing: an absolute https target sent with x-forwarded-proto: http is told https, the target's scheme, and the service is sent http, where HAProxy answers 400; an HTTP/2 request with :scheme https sent with x-forwarded-proto: http is told https by the Envoy output and http by nginx and HAProxy, a declared difference, and the service is sent http; and an x-forwarded-proto of ftp reaches the service as sent, while the Envoy output tells the check http, the connection's scheme, and nginx and HAProxy tell it ftp, a declared difference, which nginx's list of ways it is less strict names. Each row runs on full, portable, nginx-auth, haproxy-file and nginx-connection, and the rows whose keys hold no client_address on two-checks, nginx-checks and nginx-tracing too.
timeout auth.timeout, auth.when_unavailable, timeouts.idle The check holds the request for 3 s, past auth.timeout's 1500 ms, and its 401, and its 200, send their body of 100 bytes a byte every 100 ms, so it ends after 10 s. The request reaches the service with the mark of a request the check could not answer under allow, and is answered 403 under deny, on two-checks and nginx-checks, after at least 1.45 s and in under 2.95 s, 50 ms inside the two times because the row reads a different clock from the proxy's and the check's (below). Three rows, on full, portable, haproxy-file, two-checks, nginx-checks and nginx-auth. With timeouts.idle, 2 s on nginx-file, nginx-auth and haproxy-traffic and 90 s on full and portable, three rows more. In "a service silent past the idle time" the stub holds its answer for 3 s (x-stub-delay: 3s), and Envoy and nginx answer 504 as text/plain, Envoy's own text where nginx's own page would be text/html, after at least 1.95 s and under 2.95 s (measured at 2.002 to 2.005 s on Envoy 1.40.2 and on nginx), where full and portable serve it with 200 after 2.95 s or more. In "an answer that pauses less than the idle time" the stub sends the status and headers, then a space after each of two pauses of 1.25 s (x-stub-trickle: 2x1250ms), then its echo: each pause is 0.75 s under the idle time and the answer ends 0.5 s past it, so a limit on the whole answer would end it, and every proxy serves it with 200 after at least 2.45 s. In "a request body that pauses past the idle time" a POST with Content-Length: 4 is sent by hand, ab, then cd after 3 s: Envoy answers 408 after at least 1.95 s, so the row shows the idle time ended it, where nginx closes the connection with no answer, a declared difference on timeouts.idle, and full and portable serve it with 200 and the body abcd. Each bound keeps 50 ms from the time it tests, for the two clocks, and the 504 comes within 1 s past the idle time, for a loaded runner.
retry routes[].retries, routes[].rpc.retries A request that reached the service is never tried again: a first try the service closes before any answer gives 502, a gRPC call whose first try is reset gives grpc-status 14, and a 503 from the service is passed on from its first try. A request whose connection a backend refused is tried again on the other: with one of the two shop backends refusing connections, 40 requests to the shop route and 40 gRPC calls to shop are each answered by the other backend; on nginx-file, whose shop route writes retries: 0, each of the 40 requests to the shop route is answered 502 with no service, as no other backend is tried, or by the backend that is up, and at least one of each, so a proxy that answered every request 502 would fail the row. On nginx-plain, whose rpc entry retries once and sends example.hold.* to the two backends that take no connection, a call is answered 200 with grpc-status 14 after at least 9.95 s and under 10.95 s, the two tries' connect time of 5 s each, and a request to its route /held/, which retries once to the same backends, 502 in the same bounds, which a script that found the route's time spent after the first try (504 at 5 s) or a missing copy for the second try (500) fails; on the other files, which list no such method and no such route, both reach the shop route's service. A gRPC call that is not POST, a GET with gRPC's content type over HTTP/1.1 whose first try the service closes, is answered 200 with grpc-status 14 and not tried again. nginx-auth, nginx-checks and nginx-plain, whose shop routes write no retries, get nginx-file's answers there. The rows of a retried route send to /retried/products/1, the route /retried/ of nginx-file, nginx-auth and nginx-checks and the shop route of full, which retries; nginx-plain serves it by its shop route, which tries each request once, so there each answer is 502 or the other backend's, and at least one of each. nginx reads whole before the first try of a retried route a body with a length of at most 1 MiB and a request with no body, and sends any other, one in chunks among them, as it arrives and tries it once (nginx). With one shop backend refusing connections, 40 requests are each answered by the other, and 40 POSTs with neither Content-Length nor Transfer-Encoding; of 40 GETs whose 180,000 bytes come in one-byte chunks, each reaches the other backend whole through Envoy, which tries it again, and nginx answers 502 each one the refusing backend drew and serves the others, a declared difference; 40 PUTs of 1 MiB each reach it whole; of 40 PUTs of 1,048,577 bytes, one more than Envoy's buffer for a retry, at least one is answered 507 by Envoy and 502 by nginx, which tries it once, a declared difference; of 40 chunked PUTs, and of 40 over HTTP/2 with no content-length, at least one reaches the other backend with its body through Envoy, which tries again a body under its bound, and at least one is answered 502 by nginx, a declared difference. A row that needs the refusing backend drawn, or each backend, sends 40 requests, so a backend drawn at random is missed with a chance of 2^-40. A PUT whose first try the service closed once it arrived is answered 502 and not tried again, over HTTP/1.1 and over HTTP/2. A request whose every backend refuses is answered 502 and writes one upstream-failure line, with the flags URX,UF, its route and the status 502, on Envoy and on both nginx versions; the line carries no request id, so the row finds it by route and status among the lines the proxy wrote after the row began. A PUT of 1,048,577 bytes with no backend refusing is served once, whole. Nineteen rows on full and nginx-file, fifteen on nginx-auth, which writes no rpc retries, fourteen on nginx-checks, which writes no log either, and eighteen on nginx-plain, which writes no log, and fifteen on haproxy-traffic, whose rpc entry writes no retries: portable writes no retries.
gRPC routes[].rpc, routes[].rpc.to, routes[].rpc.to.<package pattern>, routes[].rpc.to.<package pattern>.service, rpc.methods, rpc.methods[].service, rpc.methods[].method, rpc.methods[].action, rpc.methods[].reports, services.<service name>.grpc, auth.ask, auth.when_unavailable, log A method of a package mapped to shop reaches shop's gRPC port over HTTP/2 and answers grpc-status 0; a method of any other package reaches accounts; the check is told a method's action and reports, and not an action the client sent, and the service receives neither; a gRPC service's X-Accel-Redirect is passed on, not followed; a call through a .. segment reaches the method's path; a call whose service refuses the connection on every backend is answered 200 with grpc-status 14, content-type application/grpc and no grpc-message; a call whose service resets its stream is answered the same, with Envoy's and HAProxy's grpc-message naming the reset, where nginx sends none, a declared difference, and a request that is not a gRPC call, sent to that route, 503 with Envoy's text, where nginx and HAProxy answer 502 with no body, a declared difference; a call the check denies is answered in gRPC's form, 200 with grpc-status 16 and the denial's body in grpc-message, percent-encoded where it holds a %, a letter outside ASCII or a line feed. A call the check denies with 403 is answered 200 with grpc-status 7; a call the check cannot answer goes on with the unchecked mark under allow, and is answered 200 with grpc-status 7 under deny; a call whose check's answer cannot be read is a check that could not answer; and a call the check cannot answer logs an unanswered check with the method's action. A call whose accepted header the check sends empty reaches the service with that header empty, where nginx sends a gRPC service none, a declared difference. Sixteen rows: all on full, portable and nginx-auth, fifteen on haproxy-file, which writes no log, eleven on two-checks and nginx-checks, which write no rpc.methods[].action, and seven on nginx-file and nginx-plain, which write neither that nor auth.ask.
path normalisation none: it is written for every file (Written with no key) .., ., //, %2F and %2e%2e in a path are resolved before a route is picked, and the service hears the normalised path; so are .. followed by an escaped slash (..%2f), by a backslash written as %5C, and by a written backslash; ..; and .;, written with dots, resolved as .. and ., which nginx and HAProxy answer 400, a declared difference; an exact path reached through ... An invalid escape, %zz, is passed on to the service, whose own 400 answers it, and nginx answers it 400 itself, a declared difference. An escaped + and an empty query reach the service as sent; an empty query after a path that needs normalising reaches it after the normalised path, and nginx answers it 400, a declared difference. A path that still holds a dot segment with a ; parameter after normalisation answers 400 and reaches no service, the ; written or escaped once: /catalog/%2e%2e;/products/1; ..%3B and ..%3b; .%3B and %2e%3b; dots written escaped, in either case of hex, before an escaped ;; an escaped ; with a value or twice over; on the last segment and on the first; between backslashes, written or as %5C; and before an escaped slash. These reach the service as sent: a ; parameter on an ordinary segment, on a name and on a name that starts with a dot; a ; and an escaped ; after three dots; an escaped ; after a name that ends in two dots and inside a name; a ; escaped twice after ..; a query that holds /..;/ or /..%3B/; and an escaped ? after ... A dot segment with an escaped ; that a .. after it removes reaches the service resolved, and nginx answers it 400, a declared difference. A path with an empty segment before a dot-dot segment, /catalog//../products/1, reaches catalog as /catalog/products/1 through Envoy and HAProxy, where nginx answers 400, a declared difference, and so does the same path with the slash escaped, /catalog/%2F../products/1, and over HTTP/2; an empty segment after a dot-dot segment, /products/../catalog//items, reaches catalog as /catalog/items through all three. After the 400 of a dot segment with an escaped ;, sent with a body that holds a second request and then the client's next request in the same write, Envoy closes the connection and answers no next request, as nginx does, where HAProxy keeps the connection and answers the next request, a declared difference. Forty-seven rows.
the path as the client wrote it none: it is written for every file (Written with no key) A path with nothing to normalise, or with only an escape Envoy keeps, is matched and reaches the service as written: /api/x%3Bv=1/admin, /api/a%20b/admin, /u/%40bob, /l/a%20b and /api/a%7cb, in the case of hex written, where nginx's and HAProxy's own files have patterns that would take the decoded character; /x%C3%A9 is not taken by the pattern ^/x..$; an empty query on such a path reaches the service as sent. A path that needs normalising and keeps no escape is matched and sent normalised. A raw | reaches the service as %7C. nginx answers 400, each a declared difference, to a path that needs normalising and keeps an escape, /api/a%2fb%3bc; to a byte Envoy escapes beside an escape; to a byte Envoy escapes on nginx-file, which has pattern routes; to an escape and to an empty query on a route with a rewrite; and to an empty query after a byte Envoy escapes. A # in the target is answered 400 over HTTP/1.1 and HTTP/2, where HAProxy resets the HTTP/2 stream, a declared difference, and so is one to a host no environment names, before its 404. An escaped NUL, %00, in the path is answered 400, and a byte above 0x7F sent raw over HTTP/2 reaches the service escaped, %C3%A9 for é, which nginx answers 400 on nginx-file, a declared difference. Twenty-one rows.
an auth-off prefix route routes[].prefix, routes[].auth, auth.ask, auth.only_with_header, auth.accept, auth.send_request On portable, which sends /public/ to shop with auth = off before its other prefix routes: a path under it reaches the service unchecked with a token the check denies, and so does one with a repeated slash, which the service hears merged; a client's own copy of the mark of a request the check could not answer does not reach the service there either. A path that .., %2e%2e, ..%2f, ..%5C or ..\ takes out of /public/ leaves the route: with a token the check denies it gets the check's 401, and with a token the check allows it reaches the service, and the check is told the path the service hears, the normalised one; without the normalisation each would stay on the route and reach the service unchecked. ..; with a token the check denies leaves the route too and gets the check's 401, and so does an empty segment before a .., /products//../public/readme, written or, over HTTP/2, escaped, which the .. removes, where nginx answers 400, a declared difference. A dot segment with a ; parameter is answered 400 and reaches no service with no token and with a token the check allows, the ; written after %2e%2e or escaped as %3B after ... On full, which has no such route, the first and the third are answered 401 by the check its shop route asks. A client's own user, question and unchecked mark do not reach the service on that route either, and a request to it logs no check time. Twenty-two rows, on full, portable and nginx-auth, and twenty-one on haproxy-file, which writes no log.
client keys routes[].rate_limit.by On nginx-plain, whose route /limited/ admits one request a minute for each client under client_address: forwarded, eleven requests to that route, each after the rows before it, so a 200 is a client's first request and a 429 one whose bucket a row before spent: an X-Forwarded-For entry with a port, 10.80.0.1:80, is the peer, so the next request with no entry is answered 429; an address, 10.80.0.2, is a client of its own, and the same address with spaces around it is the same client; an IPv6 address in capitals, FD02::7, is a client of its own, and fd02:0::7, the same address written another way, is the same client; the last entry of a list, 10.99.0.1, 10.80.0.3, is the client, and that entry alone is the same client; an address in brackets, one with a leading zero and an IPv6 address with a zone are the peer, whose bucket is spent. On every other file that writes a limit's by, no route takes /limited/, so each request reaches shop. Eleven rows, on full, portable, nginx-file, nginx-auth and nginx-plain.
hash services.<service name>.hash.header On nginx-file, whose service sessions, at shop's two addresses, picks a backend by the hash of x-session-id on the route /sessions/, which no rate limit holds: 20 requests with an empty x-session-id all reach one backend, an empty header being a value, and so do 20 whose header comes on two lines, a and b, and 20 whose two lines come as c then d and as d then c in turn, which reach different backends when the lines are hashed in the order they came. A value of x, 60,000 spaces and y is answered within 2 s, where nginx took about 14 s while its script trimmed the value with a regular expression; on portable, whose header size nginx reads in buffers of 49,120 bytes, nginx answers that line 400 itself, a declared difference. full and portable have no such route and send the path to shop, either backend. One value of the header sent 20 times reaches one backend every time on nginx-file and on haproxy-traffic, whose route /sessions/ hashes it too. Five rows, on full, portable, nginx-file and haproxy-traffic.
tracing tracing, tracing.provider.otel, tracing.provider.otel.sample.rate, tracing.tags.<tag name>.header, tracing.tags.<tag name>.env, tracing.tags.<tag name>.default, tracing.path_in_url, auth.ask, auth.accept, routes[].rpc, origins.<environment name>.hosts On nginx-tracing, which traces 5% of the requests by their x-request-id and sends its spans to the battery's collector, each row sends an id the draw picks, 000001f3-..., or one it does not pick, 000001f4-..., and reads from the collector the span that carries its id, or finds none, beside what the service received: a request whose id the draw picks is traced, with a traceparent marked sampled, and one whose id it does not pick is not; an id that is not a UUID is drawn by its first eight characters, and one shorter than eight is never traced; a client's sampled traceparent continues the client's trace, its span the client's span's child, over HTTP/1.1 and HTTP/2, when the draw picks the id, and is sent on not sampled when it does not; one marked not sampled is never traced, and a malformed one reaches the service as sent, a byte 0xFF and each of two lines with its tracestate lines included, with no span, over HTTP/1.1 and to a gRPC service; a traceparent and a tracestate line after a tab are read as Envoy reads them; one of another version than 00 continues the client's trace and keeps the client's tracestate, and is sent on not sampled when the draw does not pick the id; two tracestate lines reach the service as one, a tracestate holding the byte 0xFF reaches the service and the check as sent, Envoy adds an empty tracestate when the client sends no traceparent, and a client's tracestate reaches the span; the check receives the trace, a parent of another version than 00 and one marked not sampled continued, and no trace beside a malformed traceparent; a span carries the Envoy output's fields, its peer.address is the connection's whatever X-Forwarded-For says, and over HTTP/2 its http.protocol says so; its http.url holds http or https alone from X-Forwarded-Proto, in any letter case, and the host_header of the route's service; a request no route takes and a path both proxies refuse are named ingress, and %00 in the path, a fragment in the target, a refused upgrade and a request line of another scheme than http and https, which is answered 400, are not traced; a target of 256 bytes is whole in the span, and a query after characters Envoy escapes, a dot-dot segment, a client's x-envoy-original-path and a route with a rewrite each give the span the Envoy output's URL; a header tag reads the first of two lines, and every line joined on a header Envoy joins, one sent empty is not set, one absent takes its default, a tag on a header the check accepts reads the value the check accepted, and its default on a route the check is off for, an environment tag whose variable is unset takes its default, and so does one on a variable named like a property every object has, one whose variable is set reads it, and one on a set variable named like such a property reads it through the Envoy output and takes its default through nginx, a value keeps a quote, a backslash and bytes above 0x7F, and a default keeps a dollar, a quote and a backslash; a tag named like a span field replaces it, and keeps the field's value when its header is absent; the tracing headers a client sends do not reach the service; a gRPC call's service receives the span's traceparent; a User-Agent after a tab is held by the span without it; a request from a refused origin is traced under its route. nginx's own literals are the differences its rows of tracing.provider.otel, tracing.path_in_url and tracing.tags.<tag name>.header declare: a traceparent of another version than 00, or after a tab, is sent on to the service and the check with the client's version and ids and no span, and a tracestate after a tab beside a parent gives no span, a User-Agent after a tab gives no span and is sent on not sampled, a gRPC call's malformed traceparent on two lines is answered 400, a target the Envoy output normalises or rewrites, or beside a client's x-envoy-original-path, gets no span and is sent on not sampled, and the query after characters Envoy escapes is answered 400 on nginx-tracing, whose path nginx refuses, no empty tracestate goes with a traceparent of nginx's own, the status in a span is a number, a request no route takes has an empty upstream_cluster, a refused origin's span is named ingress, a target over 256 bytes gets no span, and an empty or absent header tag is set empty. On full, which traces 10% with client_address: connection to a collector its world leaves unreachable, a row reads what the service received, and nothing the draw decides.
tracing a retried request tracing, routes[].retries With shop-2 refusing connections, 40 traced requests to the retried route /retried/ are each answered by shop-1 and have one span, named retried, with upstream_cluster shop_http and the status 200, whose trace and span ids are those of the traceparent the service read on the try that served it.
rate limit routes[].rate_limit, routes[].rate_limit.requests, routes[].rate_limit.per, routes[].rate_limit.burst, routes[].rate_limit.by, auth.ask After 2.5 s with no request to the route, 60 requests to /catalog/items back to back: the first 30 on full, nginx-file and nginx-auth (20 a second and a burst of 10) and the first 20 on portable (20 a second and no burst) answer 200, and at least one of the rest 429; then, of 60 requests over the limit that carry a token the check denies and one the token filter refuses, at least one is answered 429, not 401, so the limit is applied before the token and the check; of 60 POSTs over the limit with a body of 5 bytes, at least one is answered 429, and of 60 more with the next request written after the body, at least one ends its connection, so that next request gets no answer (each row after the first sends 60, as a row of 10 that finds the bucket full, which the row before may leave it, gets no 429); of 60 gRPC calls over HTTP/1.1, told by their content type, with a body and the next request written after it, at least one ends its connection too; of 60 gRPC calls over HTTP/2, at least one is answered 200 with grpc-status 8 and content-type: application/grpc, and of 60 Connect calls, POSTs that carry connect-protocol-version, at least one 429 with content-type: application/json; another route is not limited; and after 2 s with no request to the route, of 40 requests the check denies, the first 30, or 20 on portable, are answered 401 and at least one of the rest 429, so a request the check denies spends a token. On nginx-file's route /retried-limited/, which admits 10 requests a minute for all and retries once, with one shop backend refusing connections, the first 10 of 40 requests are answered 200 and at least one of the rest 429, so a request whose first try was refused spends one token, not two and not none; full, nginx-auth and nginx-plain serve the path by their shop routes, so there at least one answer comes from the backend that is up. Ten rows: all on full and nginx-auth, nine on portable, which writes no retries, eight on nginx-file and haproxy-traffic, which write no check, and two on nginx-plain, whose route /limited/ has no burst: another route is not limited, and the row of a limited route whose first connections were refused, which its shop route serves.
cost routes[].pattern, routes[].prefix, origins.<environment name>.patterns, limits.request_headers.size A path of 14,000 bytes that 24 pattern routes read, sent with an origin a pattern covers, reaches the prefix route's service with access-control-allow-origin within 500 ms, and an Origin of 8,000 bytes that 64 patterns of two * read is answered 403 within 500 ms; the request after each is served, which only a proxy still running answers. Each stopped HAProxy before its patterns ran in steps and its origin patterns were written so they cannot backtrack (HAProxy). On nginx-file the long path is served as on the other files, nginx's header buffers reading a request line as long as the header limit (nginx), and so is a path of 61,000 bytes, about the longest nginx reads at the file's 60 KiB, a head of 61,133 bytes under the 61,376 it reads ("a path of 61,000 bytes, about the longest nginx reads at 60 KiB, that 24 pattern routes read"): every file but haproxy-file serves it with access-control-allow-origin within 500 ms, and HAProxy answers 414, a declared difference; an Origin of 6,420 bytes that the last of the 64 patterns takes reaches the service with access-control-allow-origin, where HAProxy answers 403 before the service, as it answers any admitted Origin longer than it echoes, a declared difference, and one of 6,321 bytes that the 64 read is answered 403, each within 500 ms on haproxy-file and nginx-file, the files that name an environment with its host, rows of their own because nginx's first form of the patterns refused the first and held a worker past the bound with the second (nginx). Two more Origins sit past the bound nginx sets on the origin patterns. One of 16,384 bytes that the last of the 64 patterns takes ("an Origin of 16,384 bytes that the last of 64 patterns of two * takes") and one of 16,385 bytes ("an Origin of 16,385 bytes that the last of 64 patterns of two * takes") are served and echoed within 500 ms by Envoy and answered 403 within 500 ms by nginx, which matches no Origin longer than 8,192 bytes against the patterns, a declared difference on origins.<environment name>.patterns; HAProxy answers both 431, as it answers any head that outgrows its buffer, a declared difference. A path of 53,005 bytes that the 24 pattern routes read, with an Origin of 8,192 bytes that the last pattern takes, the longest nginx matches, in a head of 61,326 bytes ("a path of 53,005 bytes and an Origin of 8,192 bytes that the last of 64 patterns of two * takes, about the longest head nginx reads at 60 KiB"), is served with the Origin echoed within 500 ms on nginx-file and nginx-auth, the files that write limits.request_headers.size and name the Origin's host, and answered 404 on full and portable. On full and portable, which name no environment with the long Origin's host, each long Origin is answered 404. Then 3,000 requests over 24 connections at once, each to a path of /k01/ to /k40/ and 1,000 random a and b, which 40 pattern routes of haproxy-file read, are each served by the prefix route's service, and the proxy's memory, read through Docker's API, grows by less than 96 MiB and 4 MiB for each CPU Docker gives a container, 144 MiB with 12: HAProxy runs a thread for each CPU, and each keeps the states of its own patterns, where Envoy's RE2 keeps a cache for each pattern. On haproxy-file with 12 CPUs, HAProxy grew past the bound on the code before the states of a thread shared one budget and with a budget of about 8 MB a thread, and stays under it with the script's, of about 1 MB. It runs on every file that writes pattern and prefix routes, which haproxy-file, nginx-file and nginx-auth, the files with the 40 routes, share with full and portable. The row logs the memory it read on every run. Last, 2,000 connections are opened as a raw request's would be, over TLS with the client certificate on the files that write tls, and each sends the 65 bytes GET /products/1 HTTP/1.1, a line break, Host: localhost, a line break, x-pad: and 15 a, one more than nginx's first header buffer of 64 bytes, and then nothing ("connections each holding a head just over 64 bytes keep the proxy's memory bounded", on full, portable, nginx-file and nginx-auth, the files that write limits.request_headers.size). The proxy's memory, read through Docker's API before the first connection opens and 3 s after the last, must grow by less than 64 KiB for each connection, 125 MiB for the 2,000, and every connection must still be open with no answer right after. In the eighteen runs whose times are given below, Envoy grew by 27.4 to 33.5 KiB a connection, median 30.6 KiB, and nginx 1.30.6 and 1.31.8 by 23.3 to 26.5 KiB, median 24.7 KiB, on nginx-auth, 27.1 to 30.7 KiB, median 28.8 KiB, on nginx-file and 48.5 to 52.1 KiB, median 51.2 KiB, on portable. One header buffer of 8 MiB in place of nginx's buffers fails the row with 2,000 connections and passes it with 500, so the row keeps 2,000. The row catches that one large buffer only where the kernel's transparent huge pages are always, as Docker Desktop's are, because the buffer then takes a whole huge page of 2 MiB; with madvise or never such a buffer costs the pages a head touches and can pass the row. On every kernel the row bounds the memory a connection holds, and the generator's own test holds the buffers' form. The row logs the memory it read. Every one runs on nginx-file and nginx-auth.
connection limit and failed backends limits.connections, services.<service name>.health On nginx-limits, whose limit is two, three requests sent at once on one HTTP/2 connection, each held 1 s by the service: nginx's own limit_conn serves two and answers the third 503, a declared difference, and the Envoy output serves all three. On full, whose limit is 4096, all three are served. Then four requests while shop-2 refuses connections, at least one served, and six right after it takes them again, each served by shop: on nginx, whose passive check took shop-2 out for 10 s, each by shop-1, a declared difference. On haproxy-limits, whose shop service has one address, HAProxy's own maxconn serves two and holds the third in its queue until one ends, so it is answered a second after the others, a declared difference. Three rows, on full and nginx-limits, and the first on haproxy-limits.
traffic keys limits.request_headers.size, limits.request_headers.count, services.<service name>.health, log A header block of 50 KiB reaches the service and one of 200 KiB is refused, and 120 header lines reach it and 200 are refused; the files of nginx's battery hold 100 lines, so they refuse 120, and on portable, whose 96 KiB nginx reads in buffers of 49,120 bytes, nginx answers the 50 KiB line 400, a declared difference. With shop-2 refusing connections, every one of 20 requests to checked, checked every second, reaches shop-1 once 3 s have passed; on full the shop route's retry gives the same answers. A request whose service answers 404 is logged; HAProxy's line, of its own log-format, holds no path, so the row finds it by its frontend, backend, status, method, version and quoted request id, a declared difference. Six rows on full and haproxy-traffic, and five, all but the health row, on portable, nginx-file, nginx-auth and nginx-tracing.

The stub reports each header the service received with its lines joined by a line break, which no value holds, and the bytes of each header whose value is not well-formed UTF-8 in hexadecimal, so a row tells one line holding a comma from two lines, and compares the bytes a service received, a byte 0xFF among them: Envoy joins the lines of some headers into one before the service reads them, and a service that reads only a header's first line reads another value when a proxy passes both. A row checks that a header came on one line with oneLine, where its value cannot be known, as when it holds the peer's address.

A header the proxy writes in place of the client's, with a value no row can know before the request, such as a new x-request-id, is checked by replaced: the row names the header with the value the client sent, and the service must receive it with another value. A header the service does not receive fails the row, as one that reaches it with the client's value does.

Where a target declares a difference that changes a row's answer, the row holds that target's own literal under by, named by the declaration's row: the row of the support table in Portability whose partial cell says the difference. That literal was measured on the target's own images. The battery fails an own literal whose row the target does not declare partial, or that no run of that target reads.

A target that refuses a key a file writes cannot render that file, so the example commits, instead of the target's configuration, the lines render prints: <target>.refused beside the proxyfile. For such a file the battery starts no proxy: it passes when render exits 1 and prints exactly those lines, and a file render writes although its folder commits a refusal fails. nginx and HAProxy each refuse a key, or a form of one, they have no feature for, so an example commits a target's configuration when that target writes every key the example writes, and its refusal otherwise; the target's column of the support table says which keys those are, and the example's folder holds the one it commits.

The world

The battery starts every address the files name on one Docker network, so each rendered file runs unchanged:

Container Address What it is
proxyfile-conformance-proxy 10.0.0.10 The proxy, with the rendered files read-only at /config, the TLS files at /etc/gateway/tls and the token CA at /etc/gateway/jwt, and the environment variables APP_REGION=eu-1 and toString=ts1, which only nginx-tracing.yaml's tags read. Its listener, port 10000, is published on 127.0.0.1 at a port Docker picks, so a run never meets a port another program holds.
proxyfile-conformance-shop-1, proxyfile-conformance-shop-2 10.0.1.11, 10.0.1.12 shop: HTTP on 8080 and gRPC on 8081.
proxyfile-conformance-catalog 10.0.2.21 catalog on 8090, an address examples/full, examples/portable and conformance/testdata/nginx.yaml leave to the resolver and the other files write themselves.
proxyfile-conformance-hold 10.0.2.30 hold on 8095, which only conformance/testdata/nginx-plain.yaml names, so it starts only in a world on a file that names its address: a listener that never accepts a connection. It runs with net.core.somaxconn at 1, and fills its queue with two connections of its own at start, so the kernel drops the opening packet of every later connection and a client's connect waits until its own time runs out.
proxyfile-conformance-hold-2 10.0.2.31 A second listener that never accepts a connection, on 8095, started with the first and as it is: conformance/testdata/nginx-plain.yaml's held service has both addresses, so a call its rpc entry retries meets one that takes no connection on each try.
proxyfile-conformance-resolver 10.0.0.2, port 53 The resolver: catalog.internal.example is 10.0.2.21, collector.internal.example is 10.0.3.40, and any other name is NXDOMAIN.
proxyfile-conformance-accounts 127.0.0.1:7070 in the proxy's own network namespace accounts, HTTP/1.1 and gRPC on one port, and the auth check auth.ask names.
proxyfile-conformance-check-1, proxyfile-conformance-check-2 10.0.3.31, 10.0.3.32 checks, the check service of conformance/testdata/two-checks.yaml and conformance/testdata/nginx-checks.yaml, and the first of conformance/testdata/nginx-connection.yaml: the stub's accounts on 7070 at each address, so a row can make one of the two refuse connections.
proxyfile-conformance-collector 10.0.3.40, port 4317 The tracing collector nginx-tracing.yaml sends its spans to, and nginx-tracing-by-name.yaml by the name collector.internal.example, as OTLP over gRPC, started only in those files' worlds, before the proxy. examples/full names the same address, which its world leaves unreachable: no request waits for it.

The network is proxyfile-conformance on 10.0.0.0/22, the smallest block that holds every one of these addresses. Each file a run renders gets a world of its own: the battery removes the first file's before it starts the second's.

One stub program, conformance/stub, plays every backend, the auth check, the resolver, the backend that takes no connection and the tracing collector. The collector keeps each span a proxy exports to it and answers them all, with their ids, name, status and attributes and the type of each attribute's value, twice for a key a span carries twice, which fails the row that reads that span, on port 8080 inside its container, where the battery reads them; a row finds its span by the x-request-id its request carried. A backend answers each request with what it received: its service name, port, protocol, method, path, host and headers, so a row can say which service answered and what reached it. The check decides by the value of the authorization header (Bearer allow, Bearer deny-401, Bearer deny-403, Bearer status-503 or Bearer hang), so each row chooses the check's answer, and a row that asks the check sends x-api-key too, so two-checks.yaml and nginx-checks.yaml ask it; any other token, or none, is denied with 401, so a row that wants the check not to be asked fails when the proxy asks it. Other tokens give the answers a proxy must read with care: report, report-forwarded and report-trace allow and put in x-user-check what the check was told (x-envoy-internal and x-envoy-expected-rq-timeout-ms, x-forwarded-proto and Host, or traceparent and tracestate); big-accept allows with an x-user-id of 20,000 bytes; big-denial denies with a body of 20,000 bytes and big-header with a header of 50,000; deny-401-message denies with a body that holds a %, a letter outside ASCII and a line feed; many-headers denies with 101 header lines; status-201 answers 201; trickle and trickle-200 answer 401 and 200 with a body of 100 bytes sent a byte every 100 ms; and cut-body, length-not-a-number, two-lengths, control-byte, status-not-three-digits, no-space-before-reason, status-600, name-not-a-token, length-and-chunked, chunk-not-hex, bad-line-in-1xx and bad-trailer write an answer RFC 9112 does not read whole, or with a status of 600, as bytes net/http will not write, then close the connection; head-within and trailer-within allow with a head or a trailer section of 81,900 bytes, and head-past, trailer-past, fold-past, whose head folded lines fill, and 1xx-past, whose 1xx answer before the 200 has the long head, with one of 81,940 bytes, past the 81,920 bytes Envoy reads. fold-after-100 denies with 100 header lines and a folded line after them, allow-fold-after-100 and 1xx-fold-after-100 allow with as many in the 200 or in a 1xx answer before it, and date-twice-in-101 and cookie-twice-in-101 deny with 101 header lines, date or set-cookie on two of them. report-removed and report-settings allow and put in x-user-check what the check received of the headers a client may not send it (x-client-id, x-user-id and the mark) and of a client's HTTP2-Settings; allow-empty allows with no header, so a row sees that the service receives none of an earlier answer's; and redirect-relative answers 302 with a Location that is a path, /login?next=%2Fx. deny-401-empty answers 401 with no body; chunk-size-lf, last-chunk-lf and chunk-other-lf write a bare LF at the end of a chunk-size line, of the last chunk's line and of a chunk's data, a trailer line and the final line; folded-tab, folded-spaces and folded-twice allow with an x-user-check that the check folds onto a second line; accept-past-size allows with an x-user-check of 70,000 bytes, accept-60000 with one of 60,000 bytes, and accept-empty with one that is empty. Thirty-three more tokens write a raw answer for the forms nginx's reader of an answer reads in a way of its own, a status line, a header line or a framing that RFC 9112 does not admit, a 304, a 500, a 503 with a body, a 599 and a 101, a header name x-proxyfile-check, a denial's header that holds the byte 0xE9, and bodies of 1 MiB, of one byte more and of 1,100,000 bytes; each 200 of them reports its token in x-user-check, so a row reads back that it was allowed. A backend answers with X-Accel-Redirect set to the value of the request's x-stub-accel-redirect, on HTTP and gRPC, so a row can check that the proxy passes the header on instead of following it, and with Keep-Alive set to the value of x-stub-keep-alive, so a row can check that the proxy removes it, and with as many bytes more as x-stub-pad asks for, so a row gets an answer of 100 KB. A request the stub cannot read gets the stub's own answer, the plain-text status line of Go's net/http, and no description of the request: a row that wants that answer says unreadable, and a row that wants no service fails on it. A backend holds its answer for the time x-stub-delay names, such as 3s, and with x-stub-trickle: <n>x<time>, such as 2x1250ms, sends the status and the headers at once, then a space after each of n pauses of that time, then its echo, a space being nothing to the JSON the client reads: a row sees a service silent past a proxy's idle time, and one that keeps sending within it. A backend stops waiting when the proxy closes the request. The backends share the volume proxyfile-conformance-state: a request the stub closes on its first try is answered by whichever backend a second try reaches, so a retry would show. A backend refuses connections, as one that is down does, while a row needs it to: the battery sends its container SIGUSR1, which closes every port and connection of the stub, and waits until a connection made inside the container is refused; after the row, SIGUSR2 opens the ports again, and the battery waits until each answers. The stub is built with Go for the platform the Docker server runs and copied alone into the image proxyfile-conformance-stub.

Before the first request, the stub in accounts, which shares the proxy's network namespace, fetches the target's ready URL every 100 ms until it answers 200, for 30 s at most: the proxy's admin listener answers only on 127.0.0.1, and Envoy's image has no shell.

Each run makes throwaway TLS files: a self-signed server certificate for localhost and 127.0.0.1 with its key, a clients CA, and a copy of the server certificate as the CA bundle examples/full's auth.jwt.ca names. They are written to the folder proxyfile-conformance in the system's temporary folder ($TMPDIR, or /tmp), with the rendered file and the stub's build, and the folder is the user's alone; the proxy reads its tls and jwt folders through their own mounts. The three client certificates, one the clients CA signs, one a second CA of the same name signs, and one the clients CA signed whose validity ended a day before the run, stay in memory with their keys, so the proxy never sees them and no run writes them to disk.

What it does not prove

Running it

It needs Docker and Go. From the repository root:

go test -tags conformance -count=1 -timeout 30m ./conformance/...

-timeout 30m gives a whole run, which comes close to go test's default of 10 minutes, room to finish, and a run started while another holds the battery's lock room to wait for it (below).

The battery's tests are behind the build tag conformance, so go test ./... without it starts no container. Each row is a subtest, TestBattery/<entry>/<file>/<scenario>/<row> with _ for each space, which -v lists, so -run 'TestBattery/envoy-1.40/portable/routing' runs one scenario on one file. A row that fails prints each way its answer differs, for example service: got "shop", want "catalog", and the proxy's last 50 log lines follow.

The first run pulls the proxy's image by the digest in scripts/envoy-image, the file scripts/validate-envoy.sh also reads, so an Envoy upgrade changes one line for both; it pulls nginx's two images by the digests in scripts/nginx-images and its two images tagged -otel by those in scripts/nginx-otel-images, which scripts/validate-nginx.sh reads in the same way, and HAProxy's two by the digests in scripts/haproxy-images, which HAProxy's validator, scripts/validate-haproxy.sh, reads. With the images already pulled and a warm Go build cache, thirty runs of every entry's files, twenty-two worlds in all (one for each file of each entry that starts a proxy: eight in Envoy's entry, five in each nginx entry and two in each HAProxy entry), passed on one arm64 Mac, with Docker giving its containers 12 CPUs, and none waited for another run's lock. The runs came one after the other. Runs 11, 12 and 13 started on battery power and run 10 may have ended on it, and other work on this project, Go builds and unit tests with no container, ran on the same Mac from the end of run 10 to the end of run 21, so every time and every memory figure here is that of the eighteen runs, 1 to 9 and 22 to 30, which ran wholly on mains power with none of that work beside them. Those took 408.2 to 515.0 s, median 418.7 s, as go test times a run. Other programs used the machine during the runs: two containers of another program ran beside every run and their load was not recorded, and test containers of another program ran beside one of the thirty runs, so a run on a quiet machine may be faster. Each HAProxy entry took 12.7 to 24.3 s, median 17.6 s, each nginx entry 85.7 to 121.9 s, median 89.0 s, and Envoy's entry 201.0 to 252.5 s, median 206.6 s. A row that waits beside the rest of its scenario (above) is timed from the point the rows in sequence are done, so the row of a service that takes no connection, the retried request and the retried gRPC call whose tries take no connection, which wait in the same way, and the six rows over HTTP/2 have no time of their own here; the timeout scenario has no row in sequence, so its rows' times are their own. The cost scenario's flood took 0.16 to 0.50 s, median 0.22 s, on Envoy's files, 0.16 to 0.50 s, median 0.20 s, on nginx's and 2.47 to 4.19 s, median 2.86 s, on HAProxy's, each reading the proxy's memory twice through Docker's API in one sample. The rows that bound the time of one answer are the cost scenario's within 500 ms, which took 0.00 to 0.06 s, median 0.00 s, nginx's 400 to a path PCRE2 gives up on, on nginx-file and nginx-auth, within 50 ms, which took 0.00 to 0.00 s, median 0.00 s, the hash scenario's header of 60,000 spaces between two letters, answered within 2 s, which took 0.00 to 0.01 s, median 0.00 s, and the auth scenario's two that want a kept-alive connection answered at once, within 1 s, which took 0.00 to 0.01 s, median 0.00 s. Besides them, ten rows count on time. The rate limit's 60 requests must take under 25 ms each on average on full, nginx-file and nginx-auth, which give the catalog route a burst of 10, and under 33 ms on portable, which gives it none, because the bucket refills one request every 50 ms and 30 refills on the first three, 40 on portable, would admit every request after the first 30 or 20; the 60 took 0.02 to 0.05 s, median 0.03 s, on those three files and 0.02 to 0.07 s, median 0.03 s, on portable. The three rows of auth.timeout must be answered between 1.45 s and 2.95 s, and took 1.50 to 1.60 s, median 1.51 s, on each file and each target. Of the three of timeouts.idle, the silent service must be answered 504 between 1.95 s and 2.95 s on nginx-file and nginx-auth, whose idle time is 2 s, and served after 2.95 s on full and portable, whose idle time is 90 s; the answer that pauses less must end after 2.45 s on each; and Envoy's 408 to the paused body must come after 1.95 s on nginx-file and nginx-auth. The silent service and the paused body took 2.00 to 2.02 s, median 2.01 s, on nginx-file and nginx-auth and 3.00 to 3.01 s, median 3.01 s, on full and portable, and the answer that pauses less 2.50 to 2.53 s, median 2.51 s. A service that takes no connection must be answered 502 on nginx-plain between 4.95 s and 5.95 s, a retried request whose two tries take no connection 502 there between 9.95 s and 10.95 s, and a retried gRPC call whose two tries take no connection 200 with grpc-status 14 there in the same bounds. Two rows count on memory, the flood and the held connections (the cost scenario, above). The flood must grow the proxy by less than 144 MiB with the 12 CPUs Docker gives a container, and grew HAProxy 3.2.26 and 3.4.7 by 28 to 35 MiB, median 30.5 MiB, on haproxy-file, nginx 1.30.6 and 1.31.8 by 2 to 11 MiB, median 5 MiB, on nginx-file, by 1 to 6 MiB, median 2.5 MiB, on nginx-auth and, on portable, from a fall of 3 MiB to a growth of 3 MiB, median 1 MiB, and Envoy 1.40.2 by 66 to 72 MiB, median 70 MiB, on haproxy-file, nginx-file and nginx-auth, the files with the 40 pattern routes, and by 0 to 2 MiB, median 0 MiB, on full and portable. On full and portable the memory fell over 6 of Envoy's 36 floods, each time by under 1 MiB, and over 2 of nginx's 36 floods on portable, by 3 MiB at most. The timeout rows time the request on the client's clock, while the proxy times auth.timeout on its own, so their bounds keep 50 ms from the 1.5 s timeout and from the check's answer at 3 s; Envoy counts the timeout from the time its event loop last read, from a clock that moves in 1 ms steps under Docker Desktop's kernel, so the client can see the answer up to 1 ms before 1.5 s. A repeated row that fails says how long its requests took. The thirty runs, with every time and memory figure of this paragraph and the counts of the table below, predate the entries nginx-otel-1.30 and nginx-otel-1.31 and the two files with tracing, which add six worlds, two in each of those entries and two in Envoy's: in one run of the whole battery, which took 608 s, nginx-otel-1.30 took 55 s and nginx-otel-1.31 55 s, and Envoy's entry took 56 s on nginx-tracing and 15 s on nginx-tracing-by-name, which took 15 s and 16 s in the other two.

How often each of those rows runs in a run, and in each entry, counted from a log:

Rows In a run Envoy's entry Each nginx entry Each HAProxy entry
The cost scenario's flood 13 5 3 1
The held connections 10 4 3 0
The cost rows within 500 ms 69 27 17 4
nginx's 400 to a path PCRE2 gives up on, within 50 ms 4 0 2 0
The hash scenario's header of 60,000 spaces, within 2 s 7 3 2 0
The auth scenario's two rows within 1 s 32 12 6 4
The rate limit's 60 requests 10 4 3 0
The three rows of auth.timeout 48 18 9 6
The three rows of timeouts.idle 30 12 9 0
A service that takes no connection, on nginx-plain 3 1 1 0
A retried request whose tries take no connection, on nginx-plain 3 1 1 0
A retried gRPC call whose tries take no connection, on nginx-plain 3 1 1 0

The battery removes its own leftovers when it starts and everything it made when it ends but its lock (below): the containers named proxyfile-conformance-<role>, the network proxyfile-conformance, the volume proxyfile-conformance-state, the image proxyfile-conformance-stub and the folder proxyfile-conformance in the temporary folder. It touches no other Docker object. A run stopped before its end, by go test's -timeout or an interrupt, leaves all of them: the containers keep running, the proxy still published on 127.0.0.1, and the folder keeps the run's server key, made for that run alone and trusted by nothing else. The next run removes them when it starts; to remove them at once:

docker rm -f proxyfile-conformance-accounts proxyfile-conformance-proxy proxyfile-conformance-shop-1 \
  proxyfile-conformance-shop-2 proxyfile-conformance-catalog proxyfile-conformance-check-1 \
  proxyfile-conformance-check-2 proxyfile-conformance-resolver
docker network rm proxyfile-conformance
docker volume rm proxyfile-conformance-state
docker image rm proxyfile-conformance-stub
rm -rf "${TMPDIR:-/tmp}/proxyfile-conformance"

The names are fixed, so a run takes a lock, an exclusive flock on /tmp/proxyfile-conformance.lock, before its first file touches Docker, and gives it back once, after it removed what it made for its last file. The path is the same whatever TMPDIR is, so two runs on one machine take turns, from two checkouts, two users or two jobs. A run that finds the lock held logs that it waits, and waits once: until go test's -timeout would leave under 12 minutes for its worlds after the wait, because that timeout ends the run without removing them, or with -timeout 0 as long as the lock is held; then it stops and says another run holds the lock. With the command's -timeout 30m a run waits out a first run that ends within 18 minutes; under go test's default of 10 minutes it waits for none. A run of the checks alone, such as -run 'TestBattery/none', takes no lock. The file stays, so every run locks the same one. A killed run holds no lock, because the kernel gives it back, and the next run removes the Docker objects the killed one left. The lock does not reach past the machine's own /tmp: runs that share a Docker daemon from two machines, or from a container that uses the host's daemon, do not see each other's lock and still remove each other's containers, as does a battery older than the lock. When another network already holds addresses in 10.0.0.0/22, the run stops with Docker's Pool overlaps answer and says which range it needs; it removes no network it did not make, so free the range or run on another Docker host.

The check Required of the Pre-merge workflow runs go vet -tags conformance ./conformance/... and the battery after the Envoy, nginx and HAProxy validations, whose pulls leave the images cached. The release workflow runs the battery before it builds the release, so a release whose rendered configuration answers a scenario otherwise than the format says is not published.

Adding a proxy

A proxy joins in four places, and a row changes for it only where it declares a difference that changes the row's answer (by, above):

  1. Its generator in render.Targets in internal/render/render.go: the --target name, the configuration's file name and the function. render and the command line read the targets there.
  2. Its declaration of every core key and of the two behaviours written with no key, in a file of its own in internal/format, listed in format.Targets in internal/format/support.go. A key it does not write yet is declared refused, and a form of a key it refuses, by the key's value or for want of another key, is a Refuses entry of that key, which it declares partial.
  3. go test ./internal/render -run TestExamplesAreCurrent -update writes each example's configuration for it, or <proxy>.refused where it refuses a key the example writes, and go test ./internal/format -update writes its column of the support table.
  4. An entry per image, made in a file of its own, conformance/targets_<proxy>_test.go, by a function targets in conformance/targets_test.go calls: its label, the --target name, the files it runs, the header it adds to a request the check could not answer, its image pinned by digest, its arguments and the URL that answers 200 when it is ready. Its images are in scripts/<proxy>-images, one a line, tagged with its version and pinned by digest, read by its entries through pinnedImages, which labels each <proxy>-<major>.<minor>, and by its validator, scripts/validate-<proxy>.sh, which Required runs on every configuration the examples commit for it. The rendered files are mounted in /config, the configuration at /config/<file name>, and the ports 10000 and 9901 and the folders /etc/gateway/tls and /etc/gateway/jwt are the examples', the same for every target. The files of its entries of one version together must run each row Envoy's files run, unless its target refuses one of the row's keys, or the battery fails before any container starts: an image that loads only some files, as nginx's images tagged -otel load the tracing module, runs them in an entry of its own, which is judged with the other entries of the same version.

The documentation site

The README and the pages under docs/ are also published as a site, at https://kitecorp.github.io/proxyfile/, built from the same files, so no page is written twice. That address is the line baseURL in site/hugo.toml, the one line to change when the site is served at another. scripts/build-site.sh builds it with Hugo, the release pinned by digest in scripts/hugo-image, from the configuration in site/. Each page keeps its file's place and name, capitals included, docs/<name>.md as docs/<name>.html and the README as the first page, index.html. A link between pages leads to the built page and its heading, a link to any other file of the repository leads to that file on GitHub at the commit built, and the build fails on a link to a file the repository does not hold. A page of four sections and subsections or more starts with the list of them. Each page names, for search engines and link previews, a description taken from its first paragraph, its one canonical address and its Open Graph title, description, type and address; sitemap.xml lists every page, and robots.txt names the sitemap. The README's own Made by section is the credit on the first page, and the footer of every other page renders it.

A Mermaid block, in a list or a quote too, is drawn as on GitHub, by a copy of Mermaid the build fetches at the release pinned in site/hugo.toml, from jsDelivr or, when jsDelivr does not answer, from unpkg, and refuses unless its SHA-256 sum is the one pinned there. The site serves that copy itself, and a page with a diagram asks for it only when a diagram comes near the screen. Until then the diagram's box keeps its place, and a reader without scripts sees the block as code. A wide diagram is drawn at its own size and scrolls in its box.

The build refuses a page that the site would show otherwise than GitHub, naming the page and the text. Before Hugo runs, it refuses a page that holds two opening braces followed by < or %, which Hugo reads as the start of its own shortcode anywhere in a page, in a code block too, and a page whose first text starts with -, +, { or #+, which Hugo reads as front matter. Hugo then refuses a page whose Markdown writes a script, an on... attribute such as onload, a frame, an object, an embed or a base element: GitHub never runs a script a page writes, and the site would.

scripts/check-site.sh then reads what was built, every tag as a browser reads it, in any quoting or case and outside comments. It fails when a page git ls-files 'docs/*.md' lists has no built page, when a footer is not the first page's Made by paragraph byte for byte or a page says it twice, when a Mermaid block is not drawn, when a page lacks its description, canonical address or Open Graph names or the sitemap does not list it once, when a page of four sections or more does not start with their list, when a page holds a script the theme did not write, a frame or a base element, when the browser would fetch from another host something the page's own Markdown does not name, or when a link leads to no built file or names no heading of it. A link that leaves the site is checked for its form, never fetched.

The check Required builds and checks the site without publishing it, so a change that breaks a page or a link fails before it merges. The Pages workflow builds and checks it the same way on each push to master, or when started by hand, and publishes it when it runs on master. To build and check it yourself, with Docker running:

site=$(mktemp -d)
scripts/build-site.sh "$site" && scripts/check-site.sh "$site"