proxyfile

nginx

proxyfile render --target nginx writes nginx.conf, a whole nginx configuration, from the same proxyfile as Envoy's, and proxyfile.js beside it. The two go together: nginx.conf loads nginx's njs module and imports proxyfile.js from its own folder, and nginx refuses to load the configuration when the script or the module's file is missing (measured on nginx 1.30.6 and 1.31.8). The official nginx images carry njs; a distribution's nginx may need its njs package installed. Line 2 of nginx.conf names the version it needs, njs 1.0.2 or later, the one measured. With auth.ask the file loads njs's stream module too, which the official images carry (The auth check). proxyfile check compares both files.

proxyfile.js reads what nginx's own variables cannot give the file. It reads the name and the value of every request header, and nginx.conf answers 400 when a name is outside RFC 9110's token characters, when a value holds a control byte other than a tab, when a name holds _ and nginx reads it as one of its own headers, or when a second X-Forwarded-For, X-Forwarded-Proto, X-Request-Id, Content-Type, User-Agent or Referer line comes: the section The client's address and removed headers says why for the names with _ and the second lines, and What nginx writes with no key for the values. It also reads the lines of Origin, which it joins as the Envoy output does, as The browser origins an environment admits describes. It counts the header fields of an HTTP/2 request against limits.request_headers.count (The header limits), and tells an X-Forwarded-Proto line that is present and empty from none (The client's address and removed headers). With auth.ask it also asks the check service and reads its answer (The auth check). With a rate limit or a hash keyed by the client under client_address: forwarded it reads the client from X-Forwarded-For as Envoy reads it, with hash.header it reads the lines of the hashed header, and with a rate limit it tells which form of answer a request over the limit gets (Rate limits, Backends: balance, hash and DNS names).

nginx's column in the generated table on Portability says how this version writes each key and, for a key it refuses, what to write instead. A key it does not write for nginx, or a value of a form it does not write, is refused at render by the key's name where the file writes it, so a file that needs one is never rendered as if the key were not there; the one exception is health's path, grpc, interval, timeout and healthy, which are accepted and not read (Passive health checks). A value is refused in the same way, by its key, when its nginx form is longer than nginx reads: 4094 bytes between quotes, or 4095 without them, and one byte less where a space follows the value, as after a map's key or a header's name (measured on nginx 1.30.6 and 1.31.8). auth.jwt is refused because open-source nginx verifies no token: auth_jwt is NGINX Plus's (ngx_http_auth_jwt_module).

The path a route is matched on

Envoy picks a route on the path as the client wrote it, unless the path holds //, a . or .. segment, an escaped unreserved character (a letter, a digit, -, ., _ or ~) or %2F; then it picks it on the path it has normalised, which keeps every other escape as written, in the letter case written. Before it matches, it escapes a raw ", <, >, ^, `, {, | or }, and over HTTP/2 a byte above 0x7F. The service receives the path the route was picked on, and the query as written. The nginx file reads the path as the client wrote it and does the same, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2:

nginx cannot rebuild Envoy's path for the rest, so it answers them 400 where Envoy serves them, each declared on its row: a path that needs normalising and keeps an escape (/api/a%2fb%3bc); a character Envoy escapes beside a kept escape; a character Envoy escapes in a file with a pattern route, where a pattern could take the raw character Envoy never matches; a kept escape on a route with a rewrite, because nginx rewrites the decoded path; an empty segment before a dot-dot segment (/api//../admin), because Envoy resolves the .. before it merges the slashes and nginx merges first; and an empty query, a ? with nothing after it, wherever nginx would rewrite the path, because nginx drops the ? Envoy sends on.

The browser origins an environment admits

Every file checks the browser origin, with or without cors and lists of origins, as the Envoy output does: a request or a preflight whose Origin an environment neither lists, covers with a pattern nor serves as one of its own hosts is answered 403 before a route is picked, so it reaches no service, a WebSocket upgrade included. An origin on one of the environment's own hosts is admitted on either scheme and any port, and compared as written: http://LOCALHOST, http://localhost. and null are refused where the host is localhost. A request with no Origin is not checked. A present but empty Origin is refused as the Envoy output refuses it, over HTTP/1.1 and HTTP/2. The Origin lines are read as Envoy reads them: the lines that are not empty are joined with a comma, each without the spaces and tabs around it over HTTP/1.1, and the service receives them as one line.

origins.<environment name>.origins and patterns admit more origins, and cors sets the lines a browser reads. An origin in a list, or one a pattern covers, is admitted, and the answer to its request carries Access-Control-Allow-Origin with the origin as sent, and Access-Control-Expose-Headers when cors.expose_headers is written, in place of any the service sent. A listed origin is compared ignoring case, so HTTPS://WWW.EXAMPLE.COM is admitted by https://www.example.com and sent back as written, and a pattern in the case written, so HTTPS://SHOP.EXAMPLE.COM is refused by https://*.example.com. A preflight from an admitted origin is answered 200 by nginx itself, when a route takes its path, with the Access-Control-Allow-Methods, Access-Control-Allow-Headers and Access-Control-Max-Age lines that the cors keys write (a * for the first two sends what the request asks for), and with Access-Control-Allow-Origin and Access-Control-Expose-Headers where a list holds its origin. An origin on one of the environment's own hosts that no list holds is admitted without those two lines: its request reaches the service, and its preflight carries the other three. A path the file refuses with 400, an upgrade other than WebSocket, a request no route takes and a request nginx refuses while it reads it, such as one with a NUL byte in a header, which the Envoy output refuses too, or a header line longer than nginx's header buffers read (The header limits), carry none of them. Five differences, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, which nginx's rows of origins.<environment name>.hosts, origins.<environment name>.patterns, cors.allow_methods and cors.allow_headers name:

What a pattern route and an origin pattern cost

nginx runs each pattern route and each origin pattern with PCRE2 (10.46 in both images, with no JIT), and a worker serves no other request while a run lasts. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, on an arm64 Mac, with one worker unless it says otherwise; a time is the least and the most of three requests sent one at a time, except in the first four bullets, which were measured on this version and give the median and the largest of the requests each names, sent one at a time in the first three. The other bullets were measured on the version before it, which did not yet write limits, timeouts or log:

The client's address and removed headers

With client_address: connection, which a file that writes tls takes when it writes no client_address, the file takes each client for one it faces, as the Envoy output does. Each route's location sends the service:

It also empties x-envoy-decorator-operation, x-envoy-downstream-service-cluster, x-envoy-downstream-service-node, x-envoy-original-host and x-envoy-original-path, which only a proxy in front may send, besides the x-envoy- headers every file empties. A client's own X-Real-IP and Forwarded pass, as through the Envoy output. With forwarded the file writes none of these: the client's X-Forwarded-For passes as received, and the client's X-Forwarded-Proto and x-request-id are kept when it sends them. These answers are equal to the Envoy output's, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, but for an absolute https target sent over plain HTTP under connection, which nginx serves (below), a tab before or after a value, which reaches the service where Envoy removes it (the list), and three refusals, named on nginx's request-line row of Portability, which proxyfile.js makes before a route is picked:

The configuration cannot tell the spellings or the lines apart, so proxyfile.js refuses both.

With forwarded, which a file with no tls takes when it writes no client_address, an X-Forwarded-Proto line that is present and empty is answered 404 before a route is picked, as the Envoy output answers it, whether the line is empty or, over HTTP/1.1, holds only spaces and tabs; the configuration alone cannot tell it from no line, so proxyfile.js does. The same name on a line with no colon is answered 404 too, where the Envoy output answers 400. With connection the line is not read and the request is served. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2.

With connection and no tls, an absolute https target sent over plain HTTP, such as GET https://localhost/products/1 HTTP/1.1, is served: the service receives X-Forwarded-Proto: http, and with auth.send_request: true the check is told x-forwarded-proto: http, the connection's scheme, where the Envoy output answers 403 with the body Forbidden, closes the connection and asks no check; with forwarded both serve it. It is a way nginx is less strict than the Envoy output, named in the list and on the client_address row of Portability (measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2). proxyfile 2.2.0 told the check https for it, the scheme the target names.

remove_from_clients names headers a client may not send. For each name, every route's location sets the header empty (proxy_set_header <name> "", and grpc_set_header where the location sends to a gRPC service), and every place nginx.conf itself reads a removed name reads it empty, so the request is answered as if the client had not sent it: with origin, x-request-id, x-forwarded-proto, x-forwarded-for, access-control-request-method or x-envoy-internal removed, a foreign origin was served, a preflight was sent to the service, a new request id was made, the scheme was http and no X-Forwarded-For reached the service, as with the Envoy output. A name written in another letter case, or sent twice, is removed, and a name written with _ for - is not, as with the Envoy output. nginx refuses two kinds of name when it renders:

TLS

With tls, every server of the file, the default one included, listens with TLS (listen <port> ssl, with http2 on), and nginx.conf names the certificate and its key, and with tls.clients the clients CA, at the paths the tls keys give. nginx refuses to load the file when the certificate is missing at its path (cannot load certificate, measured on nginx 1.30.6 and 1.31.8 with examples/tls/yaml/nginx.conf and no files); scripts/validate-nginx.sh makes throwaway files for the examples.

proxyfile render --target nginx refuses a tls.certificate or tls.key path that holds $, naming the key. nginx reads a $ in those two paths as the start of a variable, and a path that keeps its $ through a variable is read at each handshake, not at start, so nginx -t passes when the file is missing and every handshake then fails, where the Envoy output refuses to start (measured on nginx 1.30.6 and 1.31.8). Put the file at a path without $. The clients CA's path keeps its $: nginx reads it there as written and refuses to start when the file is missing.

The listener takes TLS 1.2 and 1.3 only. Of TLS 1.2 it takes the list the Envoy output offers, the ECDHE suites with AES-GCM or ChaCha20: with an RSA certificate that is ECDHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384 and ECDHE-RSA-CHACHA20-POLY1305, and every other TLS 1.2 cipher is refused. The three TLS 1.3 suites are the Envoy output's, the key exchange groups are X25519, P-256 and X25519MLKEM768, and ALPN offers h2 and http/1.1. A TLS server name that matches no host is served, and so, without tls.clients, is a request whose Host names another environment than its TLS server name, each as by the Envoy output. With tls.clients, the file asks for a certificate the clients CA signed (ssl_verify_client on, ssl_verify_depth 100, which is BoringSSL's default depth; the bound is not measured), and serves a good certificate, one under an intermediate sent with its chain, and a TLS 1.2 client, the service told nothing of the certificate. Four differences, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, which nginx's rows of tls and tls.clients.ca name:

The header limits

limits.request_headers sets the largest head a request may carry, the request line, the header lines and the empty line together, and the most header fields. With no limits the file holds the Envoy output's defaults, 60 KiB and 100 fields. size becomes nginx's header buffers, and count becomes max_headers:

size large_client_header_buffers
not written (60 KiB) 1 61376
4KiB 1 4032
96KiB 2 49120
128KiB 2 65504
8MiB 128 65535

Each file also writes client_header_buffer_size 64;. large_client_header_buffers takes a number of buffers, n, and their length, b: n is (size less 64) divided by 65,536, rounded up, and b is (size less 64) divided by n, rounded down. nginx reads the request into the first buffer, of 64 bytes, then moves the line that does not fit, and the lines after it, into a large buffer, and splits no line across two buffers. A head is therefore at most 64 plus n times b bytes, never more than size, so nginx never takes a larger block than the Envoy output takes for the same size. A buffer is at most 64 KiB for two costs, both measured on nginx 1.30.6 and 1.31.8. Linux's transparent huge pages back a buffer of 2 MiB or more with a whole 2 MiB page: with transparent huge pages set to always, one buffer of the whole size grew nginx by about 2,072 KiB a connection at 2, 4 and 8 MiB, where the buffers above grew it by 24 to 33 KiB. And nginx's maps read the whole line a buffer holds, so a longer buffer lets a longer path hold a worker: with buffers of 1 MiB a path of 1,000,000 bytes took nginx 202 to 231 ms, where a buffer of 64 KiB holds no line longer than 65,533 bytes and a path of 65,005 bytes took 28.79 to 29.12 ms at the median (What a pattern route and an origin pattern cost).

What each proxy reads over HTTP/1.1, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2 (the two nginx images gave the same bounds). Envoy counts the whole head and serves a head of exactly size; nginx reads less, because no line is split across two buffers:

size Envoy: largest head nginx: largest head with one long header line nginx: largest head with a long request line
not written (60 KiB) 61,440 bytes 61,438 bytes, a line of 61,372 61,376 bytes, a line of 61,336
4KiB 4,096 bytes 4,094 bytes, a line of 4,028 4,032 bytes, a line of 3,992
96KiB 98,304 bytes 49,184 bytes, a line of 49,118 49,158 bytes, a line of 49,118

One byte more, Envoy answers 431. nginx answers 400, or 414 for a request line at 96KiB and for one that passes Envoy's bound as well, a head of 61,441 bytes at the default and 4,097 bytes at 4KiB. Above 64 KiB nginx reads a line as long as one buffer, 32 KiB to 64 KiB, which is 49,118 bytes at 96KiB, 65,502 at 128KiB and 65,533 at 8MiB, where the Envoy output reads a line as long as its head allows, so the longest line does not always grow with size.

Over HTTP/2 Envoy counts the names and values of every field, the pseudo-headers included: it took 61,440 bytes at the default and 4,096 at 4KiB, and reset the stream one byte past. nginx took 61,376 and 4,032, and past them ends the connection (GOAWAY with ENHANCE_YOUR_CALM).

max_headers counts each header line, and over HTTP/2 proxyfile.js counts the four pseudo-headers and each cookie field from what nginx keeps. nginx gives :authority, a host field or both one host line, so the script counts :authority whether or not the client sent it: at the default 100, nginx served 95 fields beside :authority alone, a host field alone or both, where Envoy served 96, 96 and 95. A request over count is answered 400, where the Envoy output answers 431 over HTTP/1.1 and resets the stream over HTTP/2.

Memory held by connections that send the start of a head and wait, measured with 2,000 connections held 15 s, at the default and at size: 8MiB:

The head sent Envoy, a connection nginx, a connection
65 bytes 23.8 to 24.2 KiB 23.8 to 24.2 KiB
1,100 bytes 24.3 to 24.8 KiB 24.3 to 25.1 KiB
9,000 bytes 30.5 to 31.0 KiB 32.1 to 32.8 KiB

Held 150 s with heads of 1,100 bytes at the default, Envoy stayed at 24.1 KiB a connection and nginx at 24.8 to 25.3. In two of seven earlier runs of 145 s and 290 s nginx rose to 35.5 and 28.1 KiB a connection, at most 1.5 times Envoy's, for a reason not found; the kernel joining huge pages is the likely one, and it was not checked. The time a path or an Origin takes to read against the patterns is in What a pattern route and an origin pattern cost.

The notes of nginx's limits.request_headers.size and limits.request_headers.count rows name where nginx differs. A header block within 64 bytes of the limit can be refused, where the Envoy output takes it. Above 64 KiB, a line longer than one of nginx's buffers (32 KiB to 64 KiB) is refused, and a block of long lines can be refused under the limit. A request over size is answered 400, or 414 for a request line, where the Envoy output answers 431, and over HTTP/2 it ends the connection, where the Envoy output resets the stream. The lines of a header Envoy joins into one, such as accept, count once each, and over HTTP/2 a cookie field holding several cookies counts once for each, where the Envoy output counts it once. Two differences are less strict and are in the list below: a trailer larger than the limit is served over HTTP/1.1, and empty frames in a row are served over HTTP/2.

The connection limit

limits.connections is nginx's own limit_conn: limit_conn_zone "all" zone=pf_conn:1m; in the http block and limit_conn pf_conn <number>; in each environment's server. nginx counts a request on its way to a service while it is in process, once its header is read, each HTTP/2 stream as one and an idle connection as none, and answers one past the number 503, before its check, if any, is asked (ngx_http_limit_conn_module). An answer the file gives itself with return, such as the 404 of a host no environment names, a refused origin's 403 or a preflight's 200, is neither counted nor refused, as a return ends the request before limit_conn runs: so the default server, which answers every request that way, holds no line, and the admin server none, so that /stats, which nginx answers after that, is never answered 503. No line limits a service's connections. Above 65535 the key is refused, as nginx refuses it (ngx_http_limit_conn_module.c line 705).

The timeouts

timeouts.idle is written as per-operation timers, the body, send, read and connect times below, where the Envoy output has one idle timer; four per-operation timers stand for Envoy's one, as the note of nginx's timeouts.idle row says. timeouts.upstream_idle is the time nginx keeps an idle connection to a service. What the file writes for each, with the grpc_ twin of each proxy_ line carrying the same time:

timeouts client_body_timeout, send_timeout, proxy_send_timeout proxy_read_timeout proxy_connect_timeout keepalive_timeout in each upstream
not written 5m 15s 5s 1h
idle: 2s, upstream_idle: 3s 2s 2s 2s 3s
idle: 90s 90s 15s 5s 1h
request: none 5m 5m 5s 1h
idle: none the longest time 15s 5s 1h
request: none, idle: none, upstream_idle: none the longest time the longest time 5s the longest time

The read time is the shorter of idle and 15 s, or idle itself with request: none (5 minutes with no idle), and the connect time is the shorter of idle and 5 s. none is written as 9223372036854775807ms, the longest time both nginx images load (they refuse 9223372036854775808ms) and nginx's form of no limit. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, with idle, request and upstream_idle all none, a service silent for 6 minutes was served by all three after 359.9 s, and a service connection was reused 1, 4 and 70 s later by all three. With upstream_idle: 3s and one nginx worker, a service connection was reused after 1 s and opened again after 4 s, by all three.

A service that takes the connection and then sends nothing is answered 504 with Envoy's text, so a client reads the same body from either proxy:

A service that takes the connection and does not read the request's body: a retried route's location, which reads a body of at most 1 MiB whole before its first try, writes its own proxy_send_timeout, the read time, and each of its copies the time left, so a PUT of 1 MiB to such a service is answered 504 after 15.0 s by the Envoy output and by both nginx versions, to a service that reads one byte every 20 s as well, and through a retried route of three backends whose first two took no connection (three runs of each, 15.0 to 15.1 s). A location that streams the body, a route's with no retries or the copy of a retried route that tries once, sends it with the send time of the table; what that answers, and the line tried in its place, is in the list below. A client that stops its body for 20 s while the service reads is served by all three.

timeouts.request with a time is refused for nginx. Its time limits count the time between two reads or two writes, so no directive holds a time for a whole request; write timeouts.request: none, or render the file for Envoy. Without the key nginx answers 504 to a service silent for 15 s after the request, as the Envoy output does, and with none it ends a request after 5 minutes with nothing read or written, as Envoy does. An answer that keeps arriving runs past the 15 s at which Envoy ends a request, and so does a request body its service keeps reading slowly, on a retried route too: differences in the list below. What nginx answers when the connect time passes is in What nginx writes with no key.

Rate limits

routes[].rate_limit and the rpc entry's rate_limit are written with nginx's limit_req, one zone of shared memory for each limited entry of routes, as the Envoy output has one bucket for each: every method the rpc entry routes shares the entry's zone, and a route's copy in every environment shares the route's. For examples/limited, whose rpc entry admits 100 calls a second for every client together and whose login route admits each client 5 requests a minute with a burst of 5, the http block holds

js_set $pf_client_key pf.clientKey;
limit_req_status 429;
js_set $pf_429_kind pf.limitedKind;
limit_req_zone "all" zone=pf_rl_1_all:16m rate=100r/s;
limit_req_zone $pf_client_key zone=pf_rl_2_client:16m rate=5r/m;

and each location of the login route

limit_req zone=pf_rl_2_client burst=9 nodelay;
error_page 429 = @pf_429$pf_429_kind;

A zone's key is a constant for by: all, the peer's address, $binary_remote_addr, for by: client under client_address: connection, and under forwarded the client proxyfile.js reads as Envoy reads it: the last X-Forwarded-For entry when it is an IPv4 or IPv6 address, an IPv6 address compared by its value, and the peer otherwise, so an entry with a port, in brackets, with a leading zero or with a zone counts as the peer. The rate is <r>r/s when requests in per make a whole number a second, and <m>r/m when they make a whole number a minute. nginx admits one request and then up to burst more at once with nodelay, so its burst is requests plus burst less one, and a zone that has been quiet admits the requests + burst a full bucket of the Envoy output holds. routes[].rate_limit.per and routes[].rpc.rate_limit.per giving no whole number of requests a second or a minute are refused by --target nginx: nginx's rate is a whole number of requests a second or a minute; write requests and per so that one of them is whole.

Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, these are the same:

nginx is stricter in two ways, each declared on the row of its key in Portability in these words, for an entry of routes and for its rpc entry alike:

A zone is named by its number and the kind of its key, pf_rl_<n>_all, pf_rl_<n>_peer or pf_rl_<n>_client, because nginx refuses a reload that gives a zone another key and keeps the old configuration running with one line in its error log; so a reload that changes by, client_address, a rate or the order of the limited routes is taken (measured on nginx 1.30.6 and 1.31.8). At a reload nginx keeps a zone's counts where the Envoy output starts new buckets, and two limited routes of one key kind that change places take each other's counts until they refill.

What a limited route costs beside a plain one is in What a rate limit and a hash cost.

Retries

The rpc entry's retries is written as grpc_next_upstream error timeout; and grpc_next_upstream_tries of retries plus one in the locations of its methods' routes, so a call whose connection the service refused, or did not open in the connect time, is tried on the next backend, as the Envoy output tries it, and a call nginx sent is never sent again. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2: with one of two backends refusing, 40 calls were all served by the other, and a call whose stream or connection the service reset after it read it reached the service once through both. A call whose method is not POST, such as a GET or a PUT with gRPC's content type over HTTP/1.1, goes to a copy of its route's location without those two lines, because nginx sent such a call again after the service had read it. nginx declares two differences on routes[].rpc.retries in these words: a call whose method is not POST is not tried again; a service with one backend is not tried again, where the Envoy output tries the same backend. A third is on the list of ways nginx is less strict: a retried request or call is sent again at once and with no limit on retries at the same time.

A route's retries of 1 or more tries a request again only when nothing of it reached a service, the requests the Envoy output's connect-failure and reset-before-request conditions try again, and nginx never sends a service a request twice. nginx's own retry stays off, as proxy_next_upstream off in every file's http block, because it sends a request again after a service read it, for every method but POST, LOCK and PATCH: with one of two backends resetting or closing the connection after it read a GET or a PUT, the Envoy output delivered each request once and answered 502, where nginx with proxy_next_upstream error timeout delivered it twice. A retried route is written instead in this form:

Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, with one to eight backends that refused the connection, did not answer ARP or did not take it in time: each request was served or answered as through the Envoy output, within half a second of its time, but in the cases below, and no service read a request twice. The check was asked once for each request, every second try included, and the accepted headers and the unchecked mark reached the service on it; under a limit of 10 requests a minute, 10 of 14 requests were served and 4 answered 429, as through the Envoy output, the request a retry served charged once. A host_header, a rewrite, a target sent as written, IPv6 backends in either way of writing them, a WebSocket upgrade after a refused first try and a gRPC call with a content-length on an HTTP route were each served as through the Envoy output, and an RPC route on the same service kept its own lines.

The differences, which nginx's routes[].retries row declares in these words: a request is tried again only when its last try sent the service nothing, at once and with no limit on retries at the same time, which the less strict list names, on the next backend in the service's order rather than one the balance or the hash picks, at most as many times as the service has other backends and never more than 6; a service with one backend is not tried again, where the Envoy output tries the same backend; a service found by a DNS name is tried again on the same name, where nginx may pick the backend that failed; a request is tried again only when it has no Transfer-Encoding and a Content-Length of at most 1 MiB, when it has neither header over HTTP/1.x, which gives it no body, or when it is a GET, HEAD, OPTIONS or DELETE over HTTP/2 with no content-length, and nginx reads its body into memory before the first try, so the service gets it only once all of it arrived, and with a slow client a request whose tries fail ends up to the upload's time later than through the Envoy output, which tries while the body arrives, and can be answered 504 where the Envoy output answers 502 or serves it; such a GET, HEAD, OPTIONS or DELETE over HTTP/2 with more than 1 MiB of body is answered 413, where the Envoy output serves it; any other request, a body longer than 1 MiB, a body sent in chunks whatever its method, or one of another method over HTTP/2 with no content-length, a gRPC call's among them, is sent as it arrives and never tried again, and a first try of it that fails is answered as on a route with no retries, where the Envoy output tries again a body of unknown length under its bound of 1 MiB, holds up to 16 MiB over HTTP/2, and answers 507 to a client that sent a longer body quickly and serves one still sending; a request nginx wrote on a kept-alive connection the service had closed is not tried again, where the Envoy output tries it again on another backend. Each as measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2:

Four forms that would start the first try before the body is whole were measured on nginx 1.30.6 and 1.31.8, and none holds, so the body is read first:

What the form costs in the file and in memory, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2 on mains power with no other container busy. A retried route writes copies of its route location: the one that tries once, and one for each second the tries before a try can take, as a location's read time is fixed and nginx counts it for each try. With the default timers and retries: 1 to two backends, a route wrote 14 more locations; with retries: 4294967295, eight backends and timeouts: {request: none}, whose tries can take 30 s, 64 more; and a file with a retried route 3 more, which its routes share. The lines, locations and bytes of each file are given with retries: 0 and then with its retries; the memory, the median of 3 runs with their range, is that of the proxy at rest with the retries:

File Lines Locations Bytes Idle memory, nginx 1.30.6 / 1.31.8 Envoy output
100 routes, 99 of them retries: 1, to two backends 7,118 to 58,955 226 to 1,615 336,758 to 2,704,622 65.7 MiB (65.2 to 65.7) / 65.6 MiB (65.3 to 65.7) 23.7 MiB (23.3 to 23.9)
100 routes, each retries: 4294967295, to eight backends, request: none 7,124 to 247,020 226 to 6,629 336,973 to 11,315,021 228.9 MiB (228.9 to 229.0) / 227.9 MiB (227.9 to 228.0) 24.6 MiB (24.6 to 26.3)
4,096 such routes 278,914 to 10,101,218 8,218 to 270,365 13,258,655 to 463,512,002 8.368 GiB (8.368 to 8.368) / 8.335 GiB (8.335 to 8.335) 67.7 MiB (67.7 to 70.1)

So a retried route took about 0.45 MiB of nginx's memory with the default timers and about 2 MiB with the longest tries, beside about 20 MiB for a 100-route file and 284 MiB for the 4,096-route file with no retries. nginx loaded each file with no warning, and nginx -t read the 4,096-route file in 11.1 s on 1.30.6 (11.0 to 12.2) and 11.4 s on 1.31.8 (11.3 to 11.6), the medians of five runs. Copies at a coarser step than a second would cost less and end a request up to that step earlier than the Envoy output.

Reading the body first holds it in memory while it arrives, measured on mains power as medians with each proxy restarted before each case, of five runs each but where a figure names its own count. A PUT of 1,048,576 bytes alone took 1.96 ms on nginx 1.30.6 and 1.67 ms on 1.31.8 through a retried route, 1.35 and 1.18 ms through a route with no retries, and 1.40 ms through the Envoy output's retried route; the runs ranged from 1.07 to 2.29 ms, so no proxy was faster. 100 such PUTs at once, each sent in 64 KiB parts 100 ms apart, were all served: at their peak nginx held 97.6 MiB more than at rest on both versions (95.0 to 104.8) and about 11 MiB more 5 s later, where the Envoy output held 111.0 MiB more (109.4 to 111.6) and still held it 5 s later, and nginx opened no temporary file. 20 PUTs of 64 MiB at once, which the copy that tries once sends as they arrive, were all served with nginx holding about 10 MiB more at the peak (6.7 to 16.5, of four runs on 1.30.6 and five on 1.31.8), as on a route with no retries, where the Envoy output held 24.8 MiB more (24.6 to 25.1).

At 1,000 requests a second for 60 s on kept-alive connections, nginx served every request at that rate through a retried route with every first try served, at a median of 10.9% of a CPU on 1.30.6 and 10.0% on 1.31.8, and with every first try refused, at 13.8% and 12.8%, holding at most 27 MiB; the Envoy output used 8.1% with every first try served and 12.6% with every first try refused, and at most 30 MiB (medians of three to five runs, the runs another container was busy for left out). With every first try refused, nginx's 99th percentile was 0.49 ms on 1.30.6 and 0.45 ms on 1.31.8, where the Envoy output's was 24.4 ms (24.2 to 24.6), as it waits before a retry and answered 502 the requests over its limit of retries at the same time (the less strict list).

A reload runs the new workers beside the old ones until the old ones have ended their requests, so it needs room for about twice the memory the file holds at rest: reloading the 4,096-route file on nginx 1.30.6 at 1,000 requests a second passed a limit of 20 GiB and the kernel stopped the container (one run, on mains power).

Backends: balance, hash and DNS names

balance is written in each upstream of the service:

balance What nginx writes
round_robin, the default no line: nginx's own round robin
least_requests random two least_conn;: of two backends chosen at random, the one with fewer requests open, as the Envoy output compares two
random random;
hash with hash.header hash $pf_hash_key consistent;
hash with hash.client hash $binary_remote_addr consistent; under client_address: connection, and hash $pf_client_key consistent;, the client the rate limit reads, under forwarded

For hash.header, a map gives the header's name for each upstream, $pf_hash_name, and the script's hashKey reads the request's lines of it: an empty header is a value of its own, and the lines of one header are joined in sorted order, as Envoy joins them before it hashes them, so one value, or one set of lines in any order, reaches one backend. A request without the header goes round robin, as Envoy sends it to a backend chosen at random. The ring is nginx's own, so a value reaches one backend through either proxy, but not always the same one. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2, with three backends: round robin, least requests and random spread the requests as the Envoy output does, least requests and random by another draw of the same kind; each of 300 header values, over HTTP/1.1 and HTTP/2, and each of 300 client addresses under forwarded, reached one backend; an empty header kept every request on one backend; and with one of the three refusing, the requests that drew it were answered 502 by both, tried on no other backend.

hash on the service auth.ask names is refused by --target nginx: nginx's check relay cannot read the check's headers to hash them; write another balance on the service. That service's upstream is the relay's, in the stream block, which reads none of the http block's variables, and under client_address: connection the relay's client is nginx itself on the relay's socket, not the request's client. It is refused whether auth.enabled is true or not, so a file stays valid when the check is turned on.

A service whose address is a DNS name gets, in each of its upstreams, zone <upstream> 1m; and server <name>:<port> resolve max_fails=0; (with health, max_fails=<unhealthy>) in place of the address lines, and the http block names the server of dns.resolver, resolver 10.0.0.2:53 valid=5s;, as does the stream block of the check's relay, which looks the check's service up the same way when its address is a name. nginx asks for a name's IPv4 and IPv6 addresses both, and asks again every 5 s, as the Envoy output does; examples/dns writes two such services. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2: an address added to the name, or removed from it, was used or left within the 5 s; a name that answered no such name was answered 502, and served again once it answered; while the name server was down every request was served with its last answer; a proxy that started while no name resolved started, answered 502, and served once the names came; the check's service by name was asked at the name's address; and a name of 100 and of 1,000 addresses was asked again over TCP after the answer over UDP was cut short. nginx declares one difference on services.<service name>.address in these words: a name gives at most about 1,000 addresses for each port; the zone of 1 MiB held 1,000 addresses and not 2,000.

A DNS name with no dns.resolver is refused by --target nginx: nginx looks a name up only through the name server it is given; write dns.resolver. nginx looks a name up while it runs only through resolver, never through the host's configuration, and refuses to load an upstream that resolves a name with none.

What a rate limit and a hash cost

These figures were measured on one Mac (Apple M5 Max, Docker 29.9.3 with 12 CPUs), on mains power, with no other container above 3% of one CPU at the start and the end of each run (programs on the Mac itself also ran). They hold for nginx 1.30.6 and 1.31.8 beside the Envoy output on Envoy 1.40.2, for the configuration rendered from one file whose services each have the same three backends: a plain route to them by round robin; a route limited to 100,000 requests a second by client, which the runs never came near, so every request was counted and admitted; a service with hash.header on x-session-id, sent 300 values in turn; and one with hash.client under client_address: forwarded, sent 300 X-Forwarded-For addresses in turn. The proxies ran one at a time with every CPU of the Docker host, nginx with a worker for each. A client on the same Docker network sent each path requests alone, 20 a second on one connection for 10 s, and then 1,000 a second over up to 64 connections for 30 s, all over HTTP/1.1, and every request of the 1,000 a second was answered 200 by one of the three backends. A run gives the median and the p99 of the requests' times, the proxy's CPU 15 s into the 1,000 a second as a share of one CPU, and the proxy's memory before the first path and after the last.

Each cell is the median of five runs and, in brackets, the least and the most of the five. A figure whose five runs differ by more than three times is not given.

Path and figure Envoy 1.40.2 nginx 1.30.6 nginx 1.31.8
The plain route, alone, median 0.71 ms (0.69 to 0.77) 0.60 ms (0.58 to 0.74) 0.59 ms (0.57 to 0.70)
The plain route, 1,000 a second, median 0.22 ms (0.21 to 0.28) 0.23 ms (0.17 to 0.40) 0.21 ms (0.16 to 0.39)
The plain route, 1,000 a second, p99 0.54 ms (0.50 to 0.67) not given 0.56 ms (0.39 to 1.17)
The plain route, 1,000 a second, CPU 12.1% (8.8 to 18.3) 12.4% (7.1 to 17.8) 8.3% (6.6 to 19.1)
The limited route, alone, median 0.71 ms (0.69 to 0.73) 0.63 ms (0.61 to 0.75) 0.64 ms (0.60 to 0.69)
The limited route, 1,000 a second, median 0.22 ms (0.20 to 0.23) 0.31 ms (0.18 to 0.39) 0.23 ms (0.19 to 0.26)
The limited route, 1,000 a second, p99 0.66 ms (0.44 to 0.67) 0.62 ms (0.39 to 0.92) 0.53 ms (0.48 to 0.73)
The limited route, 1,000 a second, CPU 9.5% (7.7 to 10.7) 17.2% (8.0 to 19.0) 11.3% (9.6 to 15.1)
hash.header, alone, median 0.73 ms (0.70 to 0.75) 0.62 ms (0.61 to 0.62) 0.61 ms (0.59 to 0.75)
hash.header, 1,000 a second, median 0.23 ms (0.20 to 0.48) 0.32 ms (0.22 to 0.34) 0.20 ms (0.19 to 0.29)
hash.header, 1,000 a second, p99 not given 0.87 ms (0.52 to 1.40) 0.50 ms (0.36 to 0.63)
hash.header, 1,000 a second, CPU 10.1% (8.5 to 21.6) 17.1% (10.1 to 18.9) 10.1% (7.2 to 16.9)
hash.client, alone, median 0.71 ms (0.69 to 0.79) 0.63 ms (0.62 to 0.70) 0.62 ms (0.60 to 0.65)
hash.client, 1,000 a second, median 0.29 ms (0.24 to 0.33) 0.22 ms (0.20 to 0.40) 0.20 ms (0.19 to 0.29)
hash.client, 1,000 a second, p99 0.69 ms (0.54 to 0.78) 0.59 ms (0.48 to 0.90) 0.53 ms (0.47 to 0.63)
hash.client, 1,000 a second, CPU 12.6% (8.1 to 22.1) 13.4% (10.5 to 18.6) 12.2% (10.4 to 15.9)
Memory before the first path 21.25 MiB (20.65 to 22.45) 13.96 MiB (13.95 to 14.11) 13.91 MiB (13.85 to 14.00)
Memory after the last path 26.75 MiB (24.50 to 27.96) 22.45 MiB (22.19 to 23.21) 21.36 MiB (21.23 to 23.07)

Within each proxy, the range of the limited route and of each hash overlaps the plain route's on every figure both give, so these runs show no cost of a rate limit or of a hash above what moves from one run to the next. Between the proxies, the ranges of every time and CPU figure overlap, so those figures are level, but for two requests alone: through nginx 1.30.6 to the hash.header service in 0.61 to 0.62 ms and through nginx 1.31.8 to the hash.client service in 0.60 to 0.65 ms, where the Envoy output took 0.70 to 0.75 ms and 0.69 to 0.79 ms. nginx held less memory than the Envoy output, before the first path and after the last, in every run of both versions.

Passive health checks

With health, each server of the service's upstreams, and of the check relay's in the stream block, is written max_fails=<unhealthy>, 3 when not written, in an upstream with zone <upstream> 1m;, so the workers share one count. This is nginx's own passive check (server): a backend whose attempts fail unhealthy times within 10 s, the default fail_timeout, takes no request for 10 s. In the service's upstreams an error, a timeout and an invalid answer count, under the file's proxy_next_upstream off and grpc_next_upstream off too; in the relay's, only an error or a timeout while connecting (stream server). nginx never takes a group's one server out, so a service with one address, and each group of a retried route, writes neither line. A service with health and more than 1,000 addresses is refused, a bound under what its upstream's zone of 1 MiB holds. path, grpc, interval, timeout and healthy are not read: open-source nginx sends no check request (health_check is NGINX Plus's).

The admin port

With admin, nginx listens on 127.0.0.1 at that port and answers three paths there. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2:

None of the three paths is answered on the public listener: a route takes them as it takes any other path, on both proxies. The contents of /stats and /server_info, the status of a refused path, the answer to TRACE and the 400 to a refused absolute target are declared on nginx's admin row, and the absolute target served as a path is named there as a way nginx is less strict.

The auth check

With auth.ask the file asks your check service about a request before it goes on to the route's service, and passes an asked request on only when the check lets it through. Auth says what the check does and what it is told; this section says how nginx asks it and where its answers differ from the Envoy output's.

Three parts do it, and all three are in the two files render writes:

The check needs both of njs's modules. nginx.conf names them with load_module modules/ngx_http_js_module.so; and load_module modules/ngx_stream_js_module.so;, and the two images pinned in scripts/nginx-images, nginx 1.30.6 and 1.31.8, carry both: nginx -t accepts the rendered file on each. A file with no auth.ask, or with auth.enabled: false, loads the http module alone and has no stream block.

For the auth block of Your own service, with listen: 8000 and accounts at 10.0.0.7, proxyfile render --target nginx writes these lines, where a line that starts # ... stands for lines left out (the file has no such comment). A route location of the checked route app holds:

location @pf_r_2 {
  set $pf_log_route "app";
  set $pf_log_upstream "app_http";
  default_type "";
  js_access pf.check;
  proxy_set_header Host $pf_authority;
  proxy_set_header X-Request-Id $pf_request_id;
  proxy_set_header X-Forwarded-Proto $pf_forwarded_proto;
  proxy_set_header Origin $pf_origin;
  proxy_set_header Upgrade $pf_upgrade;
  proxy_set_header Connection $pf_connection;
  proxy_set_header HTTP2-Settings $pf_http2_settings;
  proxy_set_header TE $pf_te;
  # ... lines that set more headers empty
  proxy_set_header "x-user-id" $pf_accept_0;
  proxy_set_header "x-user-roles" $pf_accept_1;
  proxy_set_header x-envoy-auth-failure-mode-allowed $pf_unchecked;
  proxy_set_header x-gateway-action "";
  proxy_set_header x-gateway-reports "";
  proxy_pass http://app_http;
  error_page 502 = @pf_502$pf_grpc_call;
  error_page 504 = @pf_504;
}

each environment's server holds the check location:

location = /.proxyfile/check {
  if ($pf_checking = "") {
    return 418;
  }
  error_page 418 = @pf_r_$pf_route;
  proxy_pass http://unix:/run/proxyfile-8000/check.sock;
  proxy_pass_request_body off;
  proxy_set_header Content-Length "";
  proxy_set_header Connection "";
  proxy_set_header HTTP2-Settings $pf_http2_settings;
  proxy_set_header Host $pf_authority;
  proxy_set_header x-proxyfile-method $request_method;
  proxy_set_header x-proxyfile-path "/check";
  proxy_set_header x-gateway-timeout-ms 2000;
  proxy_set_header x-envoy-internal true;
  proxy_set_header x-envoy-expected-rq-timeout-ms 2000;
  proxy_set_header x-gateway-action $pf_action;
  proxy_set_header x-gateway-reports $pf_reports;
  proxy_set_header X-Request-Id $pf_request_id;
  proxy_set_header X-Forwarded-Proto $pf_forwarded_proto;
  proxy_set_header Origin $pf_origin;
  # ... lines that set more headers empty
  proxy_set_header "x-user-id" "";
  proxy_set_header "x-user-roles" "";
  proxy_set_header x-envoy-auth-failure-mode-allowed "";
  proxy_connect_timeout 2s;
  proxy_read_timeout 3s;
  proxy_buffering off;
  subrequest_output_buffer_size 1200k;
  error_page 502 504 = @pf_check_failed;
}

and the stream block follows the http block:

stream {
  js_engine njs;
  js_import pf from proxyfile.js;
  upstream proxyfile_8000_check {
    server 10.0.0.7:7070 max_fails=0;
  }
  server {
    listen unix:/run/proxyfile-8000/check.sock backlog=2147483647;
    set $pf_check_send "authorization cookie x-gateway-timeout-ms x-envoy-internal x-envoy-expected-rq-timeout-ms x-gateway-action x-gateway-reports";
    set $pf_check_ms 2000;
    set $pf_check_uri "";
    proxy_pass proxyfile_8000_check;
    proxy_next_upstream off;
    proxy_connect_timeout 2s;
    proxy_timeout 2s;
    js_filter pf.relay;
  }
}

The lines of the check location say why it is built so. The subrequest sends no body and replaces the client's own Content-Length and Connection. proxy_buffering off and subrequest_output_buffer_size let a denial of up to 1 MiB reach check from memory: with buffering on, nginx wrote such a denial to a temporary file, and with proxy_max_temp_file_size 0 instead the subrequest never ended and the client got no answer (measured on nginx 1.30.6 and 1.31.8). With these lines the check location writes no temporary file for a denial. proxy_read_timeout is one second above auth.timeout, so the handler's own timer ends a slow check first. A subrequest the relay did not answer 200 goes to @pf_check_failed, which tells check what nginx got from the relay, its status, the bytes it received and the time it took to connect, so it can tell a check that was never answered from an answer it could not read, and from a connection to the relay that nginx never made (How the answer is read).

set $pf_check_uri names the header the relay reads the path send_request tells the check from, and is empty in a file with no send_request, as here. The default_type "" line of the route location is for the denials the check sends with no body (What the client receives when the check refuses). backlog=2147483647 is the longest queue the relay's listen takes: the kernel holds the smaller of that and net.core.somaxconn, which was 4,096 in the pinned images, so the relay's queue held 4,097 connections where nginx's own default of 511 held 512, and a burst of 600 clients filled the smaller queue while the check was healthy (How the answer is read).

Which requests are asked

The check is asked about a request when all of these hold:

A request whose path still holds a dot segment with a ; parameter is asked whatever its route, as the Envoy output asks it: the route that refuses such a path is checked too (A route with auth: off). nginx writes the location @pf_r_dotparam for it, which carries the handler and, once the check lets the request through, passes the request by an internal redirect to @pf_r_refused, which answers 400 with no body and closes the connection after it. The Envoy output also closes the connection after that 400, at once when it asked the check, and answers no request sent after it, for each form that both answer 400 (measured on Envoy 1.40.2 and on nginx 1.30.6 and 1.31.8, with the check allowing); after the check's denial of the same path the Envoy output keeps the connection, 401 and then 200 on it, and the redirect that closes it is written only for the 400. Which refusals come before the check is in Which answers come before the check.

A request is asked for each HTTP/2 stream too: 100 checked streams at once on one connection were each answered by their own check (a row of the battery).

What the check is sent

The relay sends the check one HTTP/1.1 request, to the path of ask, on a new connection for each check. The path of ask goes as written, a byte of 0x80 or more in it included: with ask: checks /auth/é/ü?x=ß, Envoy 1.40.2 and both nginx versions sent the check GET /auth/é/ü?x=ß. The request holds:

The check receives well-formed UTF-8 in a header value as the client sent it, and each other byte of 0x80 or more as !. Envoy writes one ! for each byte that is not part of a well-formed UTF-8 sequence, and the relay writes the same: caf and the bytes 0xC3 0xA9 reach the check as sent, 0xC3 0xA9 0xE9 as 0xC3 0xA9 and !, and a truncated, overlong or surrogate sequence as a ! for each of its bytes (14 forms measured on Envoy 1.40.2 and on both nginx versions, with the same answer from each). A value that holds a control byte other than a tab never gets this far: nginx answers it 400 (What nginx writes with no key).

Over HTTP/2 a path byte of 0x80 or more is told to the check in x-forwarded-uri as % and two capital hexadecimal digits, as the service receives it: /products/caf\xc3\xa9\xe9 is told as /products/caf%C3%A9%E9, by the Envoy output and by nginx alike. Over HTTP/1.1 both answer such a byte 400 before the check is asked. In a file with a pattern route, nginx answers such an HTTP/2 path 400 itself, which the row of routes[].pattern declares.

The headers x-proxyfile-method and x-proxyfile-path carry the method and the path from the check location to the relay inside nginx, and the relay writes them as the request line. The check never receives them, and a client's own copies change nothing the check is told. A client's own copy of any header the proxy writes for the check, and of every header the Envoy output removes, is removed before the check is asked.

The RPC question. For each method with an action or reports, the location of that method's route sets x-gateway-action and x-gateway-reports (the names joined by commas) before the check, and writes both empty for the service, so a service sees neither. A method with neither gets no line, and a client's own copies of both are removed first.

What the service receives

When the check answers 200 and the relay reads the answer whole, the request goes on with each header an accept name names, set from the check's answer; a name the check sent twice keeps its last value, as in Envoy, and a name the check did not send is not sent to the service, whatever the client sent. The route's location sets each accepted header with proxy_set_header, from the variable check filled, beside the location's own lines for the same name.

An accepted header the check sends with an empty value reaches the service empty over HTTP/1.1, as through Envoy: the handler sets it as one space, which nginx sends as an empty value. A gRPC location sends it through a map that turns the space into nothing, because over HTTP/2 a value that starts with a space is malformed, so a gRPC service receives no such header where Envoy sends it empty, which the note on auth.accept declares.

The accepted headers count against limits.request_headers. Envoy answers 500 with an empty body when the headers the check's answer sets take the request past the size or the count of the limit, and counts the pseudo-headers and the headers it adds itself: x-request-id, x-forwarded-proto and, with client_address: connection, x-forwarded-for and x-envoy-external-address. The handler counts the same before the request goes on, with the connection's address in that mode, and answers 500 with an empty body. It errs to the strict side by a margin: measured with size: 8KiB and count: 30, nginx answered 500 15 bytes and 1 field before Envoy with client_address: forwarded, 16 bytes and 1 field before it with connection, and 30 bytes and 2 fields before it with a client's own X-Forwarded-For, and never after it, on both versions. With an absolute target, such as GET http://api.example.com:10000/products/1, Envoy counts the target's host and port as :authority whatever the Host line holds, and the handler counts the longer of the two: counting the Host line alone answered 500 23 bytes after Envoy beside a one-byte Host. The note on auth.accept declares the margin.

The handler counts only when the check's answer set a header an accept name covers, as Envoy counts only then: a 200 that sets none is served beside 96 numbered header lines by both. A chunked request's Transfer-Encoding line is not counted, as Envoy does not count it, and a request framed by its length is counted with its Content-Length line, as by Envoy: with one accepted header under client_address: forwarded, 91 numbered lines with a chunked body are served and the same lines with a body framed by its length are answered 500, through Envoy and through nginx. A gRPC call that the accepted headers take past the limit is answered as Envoy answers a gRPC call it fails itself, 200 with grpc-status: 2 and no service, where a call they take to the limit is served. Rows of the battery hold each of these on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2.

keep-alive is never set from the check's answer, as Envoy never sets it. A name written in full that nginx writes itself on the request it passes on is refused at render, and so is a name with a * prefix: the lines of the section on auth.accept below say which, and that x-forwarded-for, x-envoy-external-address and the five x-envoy- names Envoy removes from clients are refused with either client_address, so a list stays valid when the mode changes. The file removes a client's own copies of the accept names, of the two question headers and of x-envoy-auth-failure-mode-allowed before the check is asked, so a client cannot forge them. With when_unavailable: allow and no answer from the check, the request goes on with x-envoy-auth-failure-mode-allowed: true and none of the accepted headers.

What the client receives when the check refuses

Any status but 200 from the check is a refusal. The client receives the check's status, a 2xx other than 200 as 403, with every header of the answer but content-length, transfer-encoding, connection, keep-alive, upgrade and proxy-connection, and the check's body. The Envoy output passes te, trailer and a header the check's Connection names, and so does the file. A refusal to a request from a listed origin carries that origin's CORS answer, Access-Control-Allow-Origin and, when cors.expose_headers is written, Access-Control-Expose-Headers, as the Envoy output's does.

What else the client reads, measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2 with denials of these forms:

A refused request's body is never read as the next request. A client that reuses its connection sends its next request right after a request's body, and a body that holds a request would be read as the next request if the proxy answered and left the body unread. Every answer the handler writes itself goes through r.return, which makes nginx discard the request's body: the check's denial, the 403 for an unanswered check under deny and for an answer that cannot be read, a status the check sent that is passed on, and the 400 for a dot segment with a ; parameter. Measured on both versions and on Envoy, a POST whose body was a request for /products/smuggled, refused by the check, was answered 401 and nothing served the body's request, and a second request written after that body was served. r.return cannot write a 301, 302, 303, 307 or 308 with the check's body, nor a 304 as the check sent it, so for a request with a body over HTTP/1.1 it writes nginx's own page for a redirect and content-length: 0 for a 304, which auth.ask declares. For a request that has no body by its head, and over HTTP/2, whose body comes in DATA frames, those five are written as the check sent them.

A 403 of the check to a request that carries an Origin is answered with the body Invalid CORS request and content-type: text/plain in place of the check's own, for an admitted origin too; a 401 keeps its own body, and a 2xx other than 200 answered 403 keeps the check's body (measured on Envoy 1.40.2).

A request whose content-type is application/grpc alone, or application/grpc followed by +, is a gRPC call, and its refusal is answered in gRPC's form, where a plain HTTP 401 reaches a gRPC client as a transport error: 200 with content-type: application/grpc, grpc-status from the check's status (400 gives 13, 401 gives 16, 403 gives 7, 404 gives 12, 429, 502, 503 and 504 give 14, any other gives 2), the check's other headers, and the check's body in grpc-message, percent-encoded: every byte outside printable ASCII and % is written %XX, so a body of no entry: 100% sure é, a line feed and next line reaches the client as no entry: 100%25 sure %C3%A9%0Anext line.

How the answer is read

The relay passes a request on only for a 200 it reads whole as RFC 9112 writes it: a status line of HTTP/1.0 or HTTP/1.1, one space, three digits and then nothing or a space and a reason; header lines of a token name, a colon and a value with no control byte but a tab; at most one Content-Length, digits only and never beside Transfer-Encoding; and a Transfer-Encoding of chunked alone, each chunk size plain hexadecimal. It skips 1xx answers before the final one, and joins a folded header line to the value before it with all its spaces and tabs, dropping its line end, as Envoy does. It reads each head of the answer, from the status line to the empty line with their line ends, a 1xx head included, and the trailer section, to 81,920 bytes, the most Envoy 1.40.2 reads (measured to the byte), and at most 100 header lines in a head, folded lines not counted. It reads a body only to 1 MiB, and a chunk-size line only to 1,024 bytes with its line end: those two bounds are nginx's alone and are stricter than Envoy's. It holds a body for a denial alone, and reads and drops the body of a 200.

A line end is CR LF, with these exceptions that Envoy 1.40.2 makes and the relay makes with it: a bare LF ends the data of a chunk, the last line of the body and a trailer line. A chunk-size line, the last chunk's included, that ends in a bare LF is an answer Envoy reads as no answer, and the relay reads it as unreadable.

An answer it cannot read that way is not a pass. The relay tells nginx one of three things, in the header x-proxyfile-check of its own answer:

A check counts as unanswered, as it does for Envoy 1.40.2, in these cases:

A relay that closes before it writes a byte reads as an unanswered check too, the same as a check service that refuses the connection: check cannot tell the two apart, so with allow it lets the request through with the mark.

A connection to the relay that nginx never made is not an unanswered check. check reads the status, the bytes and the connect time that @pf_check_failed recorded, and takes a failed subrequest for an unanswered check only when nginx connected to the relay and the relay wrote no byte. Where nginx never connected, the check was never asked, and the request is answered 403 under allow too, as for any other failure of nginx's own parts. A burst of clients that fills the relay's queue is such a failure, and the queue is the longest the kernel allows for that reason (What the check costs gives the bursts).

An exception in proxyfile.js itself, in check or in the relay's answer, is never a pass: check answers it 403, with allow too.

The relay

The relay is the server at the end of the stream block, and the upstream above it names the check service. In the example:

The relay holds the body of a denial, up to 1 MiB, and reads and drops the body of a 200. After it has answered nginx it drops every later byte of the check.

The relay's socket

The relay listens on a socket file, /run/proxyfile-<listen>/check.sock, and the check location passes to it with proxy_pass http://unix:/run/proxyfile-8000/check.sock (for listen: 8000). nginx has no socket in Linux's abstract namespace: a listener written unix:@name is a file named @name in the folder nginx started from, and the file had the mode 0666 that nginx gives every unix listener, so that uid 65534 read the check's answer through it (measured on nginx 1.30.6 and 1.31.8). The file still has that mode, so the folder decides who can reach it.

auth.ask declares the file that a kill leaves (Where nginx answers the check otherwise than the Envoy output).

The deadline and the pass rule

check starts a timer of auth.timeout when it asks, and races it against the subrequest. With deny, a check that has not answered when the timer fires is answered 403 at auth.timeout, as the Envoy output answers it. With allow the request goes on only when the check's subrequest ends, which the relay ends at the first byte the check sends past the deadline, or after nginx's proxy_timeout of auth.timeout with no byte from it, so allow can hold a request longer than Envoy does: the note on auth.timeout below gives the bound. The relay keeps no timer of its own: njs logs an error line for a timer still waiting when a session ends. The handler clears its own timer on every path, an error included: after a refusal from its catch, nginx closed the request only when the timer fired (measured on nginx 1.30.6 and 1.31.8), and a row of the battery holds that the next request of a kept-alive client after a refused one is answered at once.

check lets a request go on only by returning with no answer, and it does so for exactly three things: a check that answered 200 and was read whole, an unanswered check under allow, and a request that is not asked. Any exception in it is answered 403 in both modes and writes no line: nginx's error log keeps emerg alone, because a request line can carry a token, and the Envoy output writes no line for a refused check.

With the check switched off

With auth.enabled: false the file writes no handler, no check location, no relay and no stream block, and loads no stream module. It still removes a client's own copies of the accept names, the question headers and x-envoy-auth-failure-mode-allowed in every route location, so a client cannot forge them, and the request log still writes authz_us as null.

Which answers come before the check

nginx answers some requests itself before it picks a route or reaches the handler. The table gives, for each kind, what the Envoy output answers and whether it asks the check first, and what nginx answers, with a token the check denies with 401 sent on every request (measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2):

The request The Envoy output nginx Compared
an Origin on none of the environment's hosts 403, not asked 403, not asked the same
a Host no environment names 404, not asked 404, not asked the same
a Host with a dot at its end 404 404 the same
an upgrade other than WebSocket 403 403 the same
a preflight 200, not asked 200, not asked the same
a preflight with a dot segment and a ; parameter, a dot or the ; escaped asked, 401 asked, 401 made the same: @pf_r_dotparam
a preflight through a dot segment and a ; parameter both written, /..;x/products/1 200, not asked: the preflight of the path it resolves asked, 401 declared on the row of the dot segment
a dot segment with a ; parameter, also under a route with auth: off, written as %2e%2e or .. asked, 401 asked, 401 made the same: @pf_r_dotparam
a backslash in the path asked, 401 400 declared on the row of the path
an empty segment before a dot-dot segment, / and . written or escaped asked, 401 400 declared on the row of the path
an escaped control character in the path asked, 401 400 declared on the row of the path
an escape Envoy keeps, on a route with a rewrite asked, 401 400 declared on routes[].rewrite
a character Envoy escapes, in a file with a pattern route asked, 401 400 declared on routes[].pattern
a path PCRE2 gives up on asked, 401 400 declared on routes[].pattern
one header of 62,000 bytes 431 400 declared on the header limits
QUERY with no Content-Type, or an empty one (over HTTP/1.1 also spaces and tabs alone), over HTTP/1.1 or HTTP/2 400, not asked 400, not asked the same
a request over a route's rate limit 429, not asked 429, not asked the same
TRACE asked, 401 405 declared on the row of the request line
TLS with no client certificate, or one of another CA the handshake ends 400 declared on tls.clients.ca
plain HTTP to the TLS port the connection closes 400 declared on tls
TLS with a Host of another environment than the server name asked, 401 421 declared on tls.clients.ca
an admin path on the public port asked, 401 asked, 401 the same
the client's own request for /.proxyfile/check asked, 401 asked, 401, routed as any other path the same
a service silent past the idle time, token denied, then allowed 401, then 504 after the check 401, then 504 the same
a service that refuses the connection, token denied, then allowed 401, then 502 after the check 401, then 502 the same
an admin path on the admin port 200, not asked 200, not asked the same
no route takes the request, and a preflight no route takes 404, not asked 404, not asked the same

Each answer nginx gives without asking, where the Envoy output asks first, is one the Envoy output serves once the check lets the request through, so nginx refuses what Envoy serves and never the reverse. The one kind where the Envoy output asks and then refuses, a dot segment with a ; parameter, nginx asks about first too, and answers 400 once the check lets the request through, closing the connection after it, as the Envoy output does.

Where nginx answers the check otherwise than the Envoy output

Each difference below is declared on nginx's row of its key in Portability. Four entries of the list of ways nginx is less strict concern the check: past the bound of local ports the auth.ask note gives, a check fails for want of a port, and with allow its request is served with the unchecked mark, so a request the check would refuse can be served; a worker with none of its 8192 connections left cannot ask the check either, with the same result under allow, as Checks in flight on few workers says; the request of a client that closes its connection while its check is asked can still reach the service once the check allows it, as the last sentence on the request line says; and under client_address: forwarded the check is told an x-forwarded-proto that is neither http nor https as it arrived, where the Envoy output tells it the connection's scheme, so a check that decides by the scheme it is told reads the word of the client or of the proxy in front. In each of the others, a request the check refuses stays refused, and nginx either refuses or holds a request the Envoy output serves, or answers or writes a line otherwise. The words are the words of the declaration.

The request log with the check

With auth.ask the request line holds one more field, authz_us, which comes first in the line's alphabetical order, and the file writes a third kind of line, as the Envoy output does (measured on Envoy 1.40.2 and on nginx 1.30.6 and 1.31.8):

What the check costs

nginx opens a new connection to the check service for each check and closes it after the answer. The Envoy output keeps its connections to the check open, and held none in TIME_WAIT in its own network namespace in any run. The figures below were measured on one Mac running Docker with 12 CPUs, on mains power, with no other container above 3% of one CPU (programs on the Mac itself, a browser and the window server among them, also ran). They hold for nginx 1.30.6 and 1.31.8 beside the Envoy output on Envoy 1.40.2, for the configuration rendered from a file that asks one check service at two addresses with when_unavailable: deny and timeout: 1500ms, with both outputs writing one request log line for each request. A rate is the answers a second, counted by a client on the same Docker network.

The rates, the latencies, the memory and the reload below were taken before the relay's socket moved to a file in a folder of its own and before the relay's queue was raised to the longest the kernel allows (The relay's socket), and were not taken again. The bursts below were taken after the queue was raised and before the socket moved. All of them were taken with 1024 connections a worker, before a file with the check wrote 8192, which holds 3.4 MiB more in each worker from its start (Checks in flight on few workers).

With a check service that closes its connection after it answers, as the battery's own stub does:

With a check service that does not close its connection right after its answer, nginx closes its connection to it first. A check that closed 20 ms or more after its answer, and one that never closed, left 50 of 50 connections in TIME_WAIT on nginx's side, on both versions, where a check that closed at once left none there, and the Envoy output kept its connection and held none. proxy_half_close on did not change which side closed first (measured on nginx 1.30.6). Each check then holds one of nginx's 28,232 local ports (32768 to 60999) for 60 s:

Clients that arrive at once, measured on mains power: 600, 1,200 and 2,400 connections opened at once, five requests on each, so 3,000, 6,000 and 12,000 requests, from a container on the same Docker network, to the Envoy output and to each nginx version in turn. Other containers ran on the Mac and their load was not recorded. The check service denied every request in the runs under allow, and allowed every request in the runs under deny.

Connections at once Under allow, a check that denies every request: Envoy served with the unchecked mark nginx served Under deny, a check that allows every request: Envoy answered 403 nginx answered 403
600 13 of 3,000 none 12 of 3,000 none
1,200 31 of 6,000 none 23 of 6,000 none
2,400 436 of 12,000 none 325 of 12,000 none

nginx 1.30.6 and 1.31.8 each answered 401 to every request under allow and served every request under deny. The Envoy output's other requests were answered as the check said: 401 under allow, and served under deny.

When the check does not answer well, each run below sent requests on 400 connections for 15 s:

Checks in flight on few workers

Each checked request in flight holds four of one worker's connections: the client's, the check location's to the relay's socket, the relay's from it and the relay's to the check. A worker with no connection left cannot open the relay's connection, which reads as an unanswered check, so with allow the request went on with the unchecked mark and the check was never asked. The Envoy output asks at most 1,024 checks at once, one on each of its check cluster's 1,024 connections, holds up to 1,024 more waiting and serves the rest with the mark: of 4,000 requests at once, its check received 1,024 in the first 200 ms and the next only as those were answered, and with a check that answers after 500 ms it served some with the mark from 1,200 at once, never asked (measured on Envoy 1.40.2). A file with the check therefore writes worker_connections 8192 and worker_rlimit_nofile 8192, so one worker carries 2,048 checks, twice what the Envoy output asks at once, and a host's lower limit of open files does not stop it first; a file with no check keeps 1024 and writes no worker_rlimit_nofile.

The table counts the requests the check would refuse that were served with the mark, the check never asked, under allow, with a check that denies each request 500 ms after it is asked: N connections opened at once, one request on each, from a container on the same Docker network, with worker_processes 2 or 4 written in place of auto in the rendered file and nothing else changed, on mains power, beside containers of other programs whose load was not recorded. Each nginx cell at 1024 gives nginx 1.30.6 and then 1.31.8; the cell at 8192 holds two runs of each version with each count, and the Envoy output's cell its three runs:

At once The Envoy output nginx at 1024, 2 workers nginx at 1024, 4 workers nginx at 8192, 2 or 4 workers
300 0 0 and 0 0 and 0 none
500 0 10 and 0 0 and 0 none
800 0 231 and 234 0 and 0 none
1,200 2 to 51 389 and 336 209 and 221 none
1,600 22 to 79 236 and 304 323 and 294 none
2,000 101 to 199 164 and 236 611 and 195 none
2,400 368 to 469 187 and 215 80 and 617 none
3,000 957 to 1,119 420 and 382 314 and 697 none
4,000 1,952 to 1,971 368 and 390 864 and 818 none

At 8192 nginx asked the check for every request and answered each 401. At 1024 it wrote 1024 worker_connections are not enough at alert for each request it could not ask, a level the file's emerg log does not show. Each worker allocates its table of connections as it starts: at 8192 each worker held 10.6 MiB at start where it held 7.1 MiB at 1024, about 500 bytes for each connection, on both versions, so the 12 workers above held 127 MiB at start where they held 85 MiB (VmRSS, read from /proc). Each worker's limit of open files was 8192, where the container's own was 1048576.

One bound is left: nginx counts a client's and a service's connections against the same 8192, so a worker with none left still serves a request unasked under allow, where the Envoy output's check keeps its own 1,024 connections whatever else the proxy carries, an entry of auth.when_unavailable on the list of ways nginx is less strict.

Reloading with checks in flight

Three nginx -s reload commands, run 10, 20 and 30 s into 40 s of checks that each took 500 ms, broke no request: each of the 1,609 (nginx 1.30.6) and 1,607 (nginx 1.31.8) requests was answered 200 after a check of its own of at least 500 ms. None was answered 502, none was refused and none reached the service without a check. This was measured with the same file and Mac as above, with when_unavailable: deny, on nginx alone. A reload under allow and a reload of the Envoy output were not measured. That run was taken before the relay's socket moved to a file (The relay's socket). After the move, one reload on each image left a request that the check allows answered 200 and one that it denies answered 401, as before it.

The request log

Without log, nginx writes one JSON line to standard output for every request. With log, it writes a line for a request whose status is status_from or higher, 600 to 999 included, for sample of the rest, and for a request the Envoy output marks with a response flag below status_from. nginx has no response flags, so a condition on the answer stands in for them; it covers the flags NR, DPE, SI, UC, UF and UT and a gRPC call, as measured. A sample of 100% writes every request, and with 0% only the status and the flags choose. The share is drawn by a hash of the request id, not by its first eight hex digits, as the note of the log.sample row says. nginx also writes the line for some requests it answers itself and Envoy serves, such as one it answers 400. The line holds no path, query, address, user agent or referrer, as the Envoy output's.

The fields are the Envoy output's but flags and details, in the same order, which is alphabetical: bytes_in, bytes_out, duration_ms, host, method, protocol, request_id, route, status, time, upstream and upstream_ms. With auth.ask the line also holds authz_us, first in that order (The request log with the check). status, bytes_in, bytes_out and duration_ms are bare numbers, duration_ms and upstream_ms are whole milliseconds, and upstream_ms is null when no service answered. A status is written with no leading zero: 0 for an HTTP/2 stream a client cut in its headers, as Envoy writes it, and 9 for an answer nginx reads with no status line, one whose status has one, two or four digits or none, where Envoy writes the status the service wrote, 2000, for four digits and 502 for the others; nginx writes that line whatever status_from says, as Envoy writes both. A status nginx reads as three other digits is written as their number, 050 as 50, 2 0 0 as 200 and 403.1 as 403, and its line only when status_from or sample takes it, where Envoy writes 502 with the flag UPE whatever status_from says. protocol is HTTP/2 over HTTP/2. A request id with a quote, a backslash and a tab in it is escaped as Envoy escapes it, "q\"b\\c\td", and bytes_out was equal on the same answer with a Content-Length (227). A service's chunked answer of 5 bytes is sent on chunked over HTTP/1.1, and nginx counts its chunk lines, 15 bytes, where Envoy counts 5; over HTTP/2 both count 5. The time is the moment nginx writes the line, 2026-11-15T13:12:45+00:00 for a request sent at 13:12:42.880 and answered 2.5 s later, where Envoy writes the request's start, 2026-11-15T13:12:40.322Z for one sent at 13:12:40.322. A start time written by proxyfile.js, or the line's time with milliseconds from a map, would cost a worker CPU on every line, so the two stay differences. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2. The differences, which nginx's log row declares in these words: the time is the moment nginx writes the line, in whole seconds with the offset of nginx's time zone (+00:00 in UTC), where the Envoy output writes the request's start in milliseconds with Z; over HTTP/1.1 bytes_out counts the chunk lines of an answer nginx sends chunked, where the Envoy output counts its body alone; an answer whose status has one, two or four digits, or that has no status line, logs the status 9, where the Envoy output logs the four digits or 502; a status that starts with 0 logs its number without the zeros (050 as 50), one with spaces between its digits the digits joined (2 0 0 as 200) and one with a fraction its whole number (403.1 as 403), each written only when that number is status_from or above or sample takes the request, where the Envoy output writes 502 whatever status_from says; bytes_in counts the whole request, not its body; upstream_ms counts from the opening of the service connection of the try that answered, not from the request's start; authz_us counts the check's time in whole milliseconds, 0, 1000, 2000 and so on, where the Envoy output counts microseconds; there are no flags and details fields; a request of a retried route that is not tried again because of its body, longer than 1 MiB or of a length nginx does not know, writes the upstream-failure flag UF after its one try failed, where the Envoy output, which tries it again, writes URX,UF; a value the Envoy output writes null is written empty, but upstream_ms; a request the origin or upgrade check refuses logs route and upstream empty, where the Envoy output names them; a request refused for a dot segment with a ; parameter logs route empty on each of its lines, where the Envoy output names the route that refuses it, refused:dot-segment-with-parameter; a request Envoy refuses as it reads it logs its host, method, request id and protocol, which Envoy writes null and HTTP/1.1; a request id or Host holding a byte above 0x7F is logged empty.

A second line, {"event":"gateway_upstream_failure","flags":...,"route":...,"status":...,"upstream":...}, is written for a request whose service failed, as the Envoy output writes it: UF for a service that refused the connection, UPE for an answer cut short of its Content-Length, for a gRPC answer whose body is shorter or longer than its Content-Length, which nginx ends itself, and for one whose head nginx refuses as malformed, and UC and UT as Envoy writes them. A call to an RPC route with retries whose last try was refused, or did not take the connection within 5 s, gets URX,UF, the flags Envoy writes for a call its retries ran out on, after one try at a service of one backend, which nginx tries no second time, after two at a service of two, and after the one try nginx gives a call that is not a POST. On a route with retries the line reads the last try too: URX,UF when every try was refused, at a service of one, two or eight backends, over HTTP/1.1 and HTTP/2 and for a PUT with a length; UT when the route's 15 s ran out, a gRPC call's too; no line when the idle time ended the request, as Envoy writes none for it; UC for a try reset after the head that followed a refused one; and no line for a request a retry served. A request the copy that tries once sent, a body over 1 MiB or of a length nginx does not know, gets UF when its one try was refused, where the Envoy output, which tries such a request again, wrote URX,UF, or answered one over its bound 507 with no line. The request line holds the upstream_ms of the try that answered (each measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2). A gRPC answer whose body matches its Content-Length gets no second line, as with Envoy, though nginx counts the frames of a gRPC answer, not its body, in $upstream_response_length; a HEAD request gets none, and a request the idle time ended gets none, as with Envoy. A service's answer whose head nginx refuses is answered 502, as Envoy answers it, for each of a space before a header's colon, a NUL in a header value, a Content-Length that is not a number, a head that ends before its blank line and a status line cut in the middle (measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2). Two cases nginx does not write, both in the list below: a chunked answer its service cuts short after a chunk, which Envoy logs with UPE and a failure line, is recorded by nginx as complete, and nginx writes neither; and an answer with a header line nginx passes on where Envoy answers 502, which Envoy logs with UPE.

Tracing

With tracing and its otel provider, nginx sends a span of each sampled request to an OpenTelemetry collector through its OpenTelemetry module, ngx_otel_module, which the official images tagged -otel carry, such as nginx:1.31.8-otel, and the plain images do not. Line 2 of nginx.conf then names the module beside njs, and the file loads it with load_module modules/ngx_otel_module.so;. On a plain image nginx -t stops at that line, with dlopen() "/etc/nginx/modules/ngx_otel_module.so" failed (... No such file or directory), and the same file loads on the image tagged -otel of the same version. A file without tracing loads no module and writes none of the tracing lines, so it loads on either image. Each behaviour this section names was measured on nginx 1.30.6 and 1.31.8, each on its image tagged -otel, beside the Envoy output on Envoy 1.40.2, and the battery's tracing rows run on both (Conformance). The xray provider is refused (What nginx refuses of tracing).

What the file writes

The spans go as OTLP over gRPC, with no TLS and no credentials, to the endpoint (otel_exporter), with service as their service.name (otel_service_name), as the Envoy output sends them.

Where nginx traces otherwise than the Envoy output

The differences, which nginx's rows in the support table declare in these words:

Tracing adds no entry to the list of ways nginx is less strict than the Envoy output: each difference above was measured and is declared on its key's row, none sends the collector, the service or the check a header or a token the Envoy output does not send, and no request waits for the collector (What tracing costs). A tab before or after a header value, such as a client's x-request-id under forwarded or a User-Agent, reaches the service with tracing as without it, one of the entries of that list.

What nginx refuses of tracing

Each is refused at render by the key where the file writes it, in these words:

What tracing costs

The figures below were measured on one Mac running Docker with 12 CPUs, so nginx ran 12 workers, on mains power, with no other container above 3% of one CPU. Each is the median of five runs with their range in brackets, unless it names another number of runs, for nginx 1.30.6 and 1.31.8 on their images tagged -otel beside the Envoy output on Envoy 1.40.2, all rendered from one file with sample.rate at 100%, so each request makes a span, with two tags, one of an environment variable and one of a header, and a route with the auth check beside one without; the proxies started afresh for each run, and a client on the same Docker network sent the requests. One proxy is said to answer sooner, hold less or lose fewer spans than another only where their ranges do not overlap. The answer times, the memory at rest and the collector's states were measured before this version changed how an env tag's variable is read, and the memory under load with it.

With a collector that answers each call at once, every request was answered 200 and every span reached the collector in each run: 201 requests one at a time, 201 one at a time through the check, then 1,000 a second for 30 s, of which nginx sent a span for each of the 30,402 or 30,403 requests and the Envoy output for each of its 30,403 and for each check. The same file without tracing ran beside it.

Envoy output nginx 1.30.6 nginx 1.31.8
Median answer, one request at a time, with tracing 1.43 ms (1.31 to 1.51) 1.02 ms (0.70 to 1.29) 1.00 ms (0.66 to 1.17)
The same without tracing 1.41 ms (1.14 to 1.47) 0.78 ms (0.67 to 1.15) 0.87 ms (0.67 to 1.15)
Median answer through the check, one at a time, with tracing 2.03 ms (1.58 to 2.12) 2.48 ms (1.62 to 2.66) 2.27 ms (1.69 to 2.62)
The same without tracing 1.80 ms (1.08 to 1.96) 2.27 ms (1.56 to 2.55) 2.47 ms (1.55 to 2.54)
Median answer at 1,000 a second, with tracing 0.21 ms (0.19 to 0.28) 0.19 ms (0.19 to 0.21) 0.18 ms (0.16 to 0.20)
The same without tracing 0.23 ms (0.19 to 0.25) 0.18 ms (0.17 to 0.29) 0.18 ms (0.17 to 0.27)
Memory at rest, with tracing 21.0 MiB (20.7 to 22.2) 57.3 MiB (57.3 to 57.4) 57.4 MiB (57.3 to 58.0)
The same without tracing 20.7 MiB (20.1 to 21.5) 50.1 MiB (50.0 to 50.2) 50.2 MiB (50.1 to 50.2)

On each proxy the answer times with and without tracing overlap, so these runs show no time that tracing adds to a request. One request at a time, on a route with no check, nginx answered sooner than the Envoy output with tracing; without it the ranges just overlap, and they overlap through the check and at 1,000 a second.

Tracing costs nginx memory under load, which it keeps. Over 300 s of 1,000 requests a second to the route with no check, each proxy started afresh with nothing sent to it before, the memory Docker counted, in MiB, was as follows, one run of each: nginx 1.31.8 and 1.30.6 with the collector asked by its address, and the file without tracing and the Envoy output, taken before with the same file and client. Both runs of nginx with tracing started with no other container above 3% of one CPU; another project's test containers used about one and a half CPUs at the end of 1.31.8's, and none was above 3% at the end of 1.30.6's. Neither version was run for 300 s through the relay, with the collector found by name, with this version's script.

At rest 15 s 60 s 120 s 180 s 240 s 300 s 120 s after the load
nginx 1.31.8, by address 58.1 208.2 216.5 219.7 222.1 223.0 222.7 223.0
nginx 1.30.6, by address 58.4 141.9 147.9 160.4 171.9 175.1 175.7 175.6
nginx 1.31.8 without tracing 51.3 55.3 55.1 56.6 56.7 57.9 57.6 55.8
nginx 1.30.6 without tracing 51.8 56.0 56.4 56.9 57.6 57.9 58.3 57.2
Envoy output, by address 22.9 25.3 25.0 25.1 24.9 24.6 25.3 25.3

nginx 1.31.8 came within 1 MiB of its highest at 150 s and 1.30.6 at 240 s, and neither gave more than 1 MiB of it back in the 120 s after the load. The module keeps at most four batches of 512 spans in each worker, its defaults, which the file keeps, and keeps each span's strings when a batch is sent, to fill them again: memory rises until each worker's 2,048 spans have held their longest values, 24,576 spans for the 12 workers here, and stays there. So the cost grows with the number of workers, one a CPU (worker_processes auto): at the end of the load each worker held 19.9 to 22.4 MiB of its own on 1.31.8 and 15.9 to 20.7 MiB on 1.30.6, against 8.4 MiB at rest. How proxyfile.js reads the tags of each request changes it. On nginx 1.31.8 asked by address, the script of proxyfile 2.2.0, which asked process.env with hasOwnProperty before it read an env tag's variable, held 523.5 MiB at 300 s (520.9 to 524.5, three runs), each worker 38.7 to 49.8 MiB; the same script asking process.env once for each traced request, with no other change, held 219.7 MiB, and with the tags read as before the script read their sources in one pass, each source on its own and the values joined in one text, 266.3 MiB (one run each). Another container was above 3% of one CPU at the start of the 219.7 MiB run, and another project's test containers at the end of one of the three runs and of both of these, up to about two CPUs. The three scripts send the same spans, so the module's batches are not the one cause of what a worker holds: why the call with hasOwnProperty left memory held in each worker, outside the script's own heap, was not found, and how much of what a worker holds now comes with the script's work for the tags and how much with the module's spans was not measured. This version asks process.env once for each traced request and takes the variable only when it is a string, which no name every object inherits gives. On nginx 1.30.6 the script of 2.2.0 was measured under load only through the relay, in one run of 900 s, so how much this version saves there was not measured. A file with the module and one proxied location alone levelled at 57.2 MiB on 1.30.6 and 57.1 MiB on 1.31.8 (one run each).

With the collector in each of five states, 201 requests one at a time and then 5,001 at 500 a second over 16 connections were sent to each proxy, each proxy started with the collector already in that state: up; an address nothing answers on; a collector that answers each call after 5 s; one that answers each call UNAVAILABLE; and one that takes each call and never answers. Every request of every run was answered 200, and no state's median is above the range with the collector up, on any proxy, so no request waited for the collector:

Collector Envoy output, one at a time / at 500 a second nginx 1.30.6 nginx 1.31.8
up 1.09 ms (0.87 to 1.54) / 0.39 ms (0.38 to 0.40) 0.98 ms (0.79 to 1.21) / 0.33 ms (0.33 to 0.45) 0.83 ms (0.74 to 1.25) / 0.34 ms (0.33 to 0.39)
nothing answers its address 1.01 ms (0.91 to 1.37) / 0.38 ms (0.36 to 0.54) 0.83 ms (0.74 to 1.10) / 0.33 ms (0.32 to 0.34) 0.75 ms (0.71 to 1.19) / 0.34 ms (0.33 to 0.39)
answers after 5 s 1.06 ms (0.90 to 1.37) / 0.39 ms (0.32 to 0.46) 0.82 ms (0.70 to 1.01) / 0.34 ms (0.32 to 0.36) 0.97 ms (0.69 to 1.06) / 0.34 ms (0.33 to 0.36)
answers UNAVAILABLE 1.05 ms (0.79 to 1.12) / 0.39 ms (0.36 to 0.40) 0.73 ms (0.62 to 1.08) / 0.33 ms (0.32 to 0.34) 0.70 ms (0.63 to 0.99) / 0.34 ms (0.33 to 0.40)
never answers 0.90 ms (0.85 to 1.08) / 0.38 ms (0.37 to 0.41) 0.80 ms (0.71 to 1.04) / 0.33 ms (0.33 to 0.36) 0.73 ms (0.70 to 1.03) / 0.33 ms (0.32 to 0.34)

With the collector up it received 5,202 spans of 5,202 from each proxy in each run. The slow collector and the one that answers UNAVAILABLE each received all 15,606 spans of the three proxies in each run, and the one that never answers 13,640 to 15,513 of them, so a proxy goes on sending to a collector that holds its calls. At 500 a second nginx answered sooner than the Envoy output, with the ranges apart, with the collector that never answers on both versions and, on 1.30.6, with nothing at the address and with UNAVAILABLE; elsewhere the ranges overlap.

Through the relay, with the collector found by name, nginx answered 1,000 requests a second for 30 s to the route with no check at a median of 0.23 ms (0.18 to 0.32) on 1.31.8 and 0.20 ms (0.18 to 0.23) on 1.30.6, and at the 99th percentile in 0.43 ms (0.34 to 0.65) and 0.38 ms (0.34 to 0.45); the same file asking the collector's address answered at 0.23 ms (0.21 to 0.23) and 0.24 ms (0.20 to 0.31), and in 0.42 ms (0.40 to 0.44) and 0.46 ms (0.37 to 0.61), three runs of each. The ranges overlap, so these runs show no time the relay adds to a request. Every request was answered 200 and every span reached the collector.

Through the relay, with the collector found by name, the five states got 1,000 requests a second for 30 s, one run on each version: every request was answered 200, at a median of 0.18 to 0.32 ms, and the collector up received every span. The relay ends a call after 10 s, so the module went on sending to the collector that never answers until the load ended, 17,586 of 30,001 spans in 43 calls on 1.31.8 and 17,350 on 1.30.6. Asked by its address, that collector got 32 calls from each version, two to four on each of the 12 workers' connections, none of which came back, and no call after the first 15 s: 13,400 and 13,024 spans.

With the collector answering each call after 5 s, 2,000 requests a second for 10 s were each answered 200, at a median of 0.19 ms (0.18 to 0.23) through the Envoy output, 0.20 ms (0.19 to 0.24) through nginx 1.30.6 and 0.20 ms (0.19 to 0.22) through 1.31.8, and each proxy dropped the spans it could not send in time. Of the 20,000 or 20,001 spans of a run, counted at the collector within 40 s of the last request, the Envoy output lost 9,770 (9,765 to 9,775), sending the rest in 2,052 calls of at most five spans, and nginx 1.30.6 lost 3,417 (2,301 to 5,232) and 1.31.8 4,494 (3,561 to 5,048), sending the rest in 34 to 41 calls of at most 512: nginx lost fewer than the Envoy output in every run.

A collector found by name, through the relay, with dns.resolver naming a name server that gives the name with a time to live of 5 s: each proxy got 50 requests a second over 4 connections, with a collector of its own name, three runs of each of two events. When the first collector left the network 20 s in and the name stopped resolving, and 30 s later a second collector started at the name, the spans reached the second 6.78 s (6.78 to 6.79) after it started through nginx 1.31.8, 8.47 s (8.44 to 8.48) through 1.30.6 and 0.19 s (0.17 to 0.19) through the Envoy output; nginx lost 1,751 to 1,753 of a run's 17,501 spans and the Envoy output 1,520, of which 1,500 are those of the 30 s with no collector. When the name moved to a second collector with the first still up and answering, the spans reached the second 7.02 s (7.00 to 7.02) after the move through nginx 1.31.8, 8.69 s (8.68 to 8.71) through 1.30.6 and 0.43 s (0.43 to 0.44) through the Envoy output, and no proxy lost a span: the first collector received 250 or 251 more from each nginx and 20 from the Envoy output, and the second the rest of the 16,001. Every request of every run was answered 200, at a median of 0.75 to 0.93 ms through nginx and 0.92 to 1.13 ms through the Envoy output. The relay asked the name server every 6 s and the Envoy output every 5 s, and the module sends the spans it holds every 5 s, so nginx's spans reach a new address some seconds after the Envoy output's.

What nginx writes with no key

Each of these is in every file the nginx generator writes, so no row of that table holds it, and each differs from the Envoy output but the answers to a service that takes no connection or sends nothing, which are here to say what nginx answers and when. Each was measured on nginx 1.30.6 and 1.31.8, and Envoy's answers on Envoy 1.40.2, unless it says otherwise.

Where nginx is less strict than the Envoy output

These differences are not stricter than the Envoy output: in each, nginx serves a request the Envoy output refuses or ends, sends a service or a client what the Envoy output would not, or leaves out a log line the Envoy output writes. proxyfile render --target nginx counts them on standard error after it writes the files, and nginx's column in Portability names each on its row. Measured on nginx 1.30.6 and 1.31.8 beside Envoy 1.40.2: