Ingresses written for ingress-nginx¶
wxingress reads the nginx.ingress.kubernetes.io/* annotations of ingress-nginx
with the semantics of ingress-nginx, so that an Ingress written for it behaves
the same here. The code is in internal/configs/ingress_nginx*.go and
internal/configs/canary.go.
General rules, as in ingress-nginx:
- An invalid annotation is ignored, with a warning event on the Ingress; values are trimmed line by line first.
- Where an annotation takes nginx variables (
rewrite-target,x-forwarded-prefix,permanent-redirect,temporal-redirect,upstream-vhost,app-root,proxy-redirect-from/-to,auth-url,auth-signin,auth-cache-key,auth-proxy-set-headers,mirror-target,proxy-ssl-name,proxy-cookie-domain/-path,upstream-hash-by,fastcgi-params-configmap), only variables every server knows are taken:$1to$9(the captures of the path's regular expression), the request's$http_<header>,$arg_<name>and$cookie_<name>, and$args,$binary_remote_addr,$content_length,$content_type,$document_root,$document_uri,$escaped_request_uri,$fastcgi_path_info,$fastcgi_script_name,$host,$hostname,$https,$is_args,$proxy_add_x_forwarded_for,$proxy_host,$proxy_port,$query_string,$realpath_root,$remote_addr,$remote_port,$remote_user,$request_filename,$request_id,$request_method,$request_uri,$scheme,$server_addr,$server_name,$server_port,$server_protocoland$uri. A value with another variable ($best_http_hostincluded, which wxingress does not define), a$without a name or a brace outside${name}where the value is unquoted is invalid: an unknown variable would makenginx -tfail for every Ingress. Where an invalid value denies the locations (auth-url,auth-proxy-set-headers,fastcgi-params-configmap), it does so here as well. - A regular expression nginx compiles (a
use-regexorrewrite-targetpath, a~ofproxy-redirect-from,proxy-cookie-pathorproxy-cookie-domain) must be one PCRE takes: .NET's balancing groups ((?<a-b>...),(?<-a>...)), group names starting with a digit, conditions that are no group ((?(text)yes|no)),\p{IsBlock}and a quantifier other than{n}in a lookbehind are refused; an Ingress with such a path is rejected. - Where an
nginx.org/*annotation sets the same, thenginx.org/*one wins, with a warning. - Where ingress-nginx refuses to serve a location because its auth cannot be
set up (a missing auth Secret, an invalid
auth-url, ...), the locations of the Ingress answer 503 instead of serving without the auth. - An annotation of ingress-nginx that is not listed below is ignored with a
warning event on the Ingress (
... is not supported by wxingress and is ignored), typos included. Those that restrict who gets an answer make all locations of the Ingress answer 503 instead, with a warning event, since serving without them would serve what ingress-nginx refuses:enable-modsecurityandenable-owasp-core-rules(other thanfalse),modsecurity-snippetandglobal-rate-limit(other than 0).enable-global-authis only ignored with a warning: wxingress has no global auth to turn on or off (the ConfigMap'sglobal-auth-*keys are not supported, below). That holds for Ingresses; on Gateway API routes only the annotations in "The Gateway API" are read, and each other one is a warning in the route's status and events. Those that restrict access make the rules of the route answer 503 instead, as the locations of an Ingress, with a warning in the log (a Policy throughwxingress.wizardtales.com/policiesrestricts a route).
Which Ingresses¶
As ingress-nginx serves the IngressClasses of its --controller-class,
wxingress serves those whose spec.controller is in -ingress-class-controllers
(chart controller.ingressClass.controllerValue and
controller.ingressClass.controllers; default nginx.wx1.eu/ingress-controller).
With k8s.io/ingress-nginx in the list it serves ingress-nginx's classes -
beside it during a migration, with controller.ingressClass.create=false so
that the chart does not take the class over. The class an Ingress names in
kubernetes.io/ingress.class wins over ingressClassName, as in
ingress-nginx. Ingresses without a class are served with
-watch-ingress-without-class (chart default on) or when a served
IngressClass is the cluster's default. A change of an IngressClass - created,
deleted, another controller - changes what is served without a restart.
Shapes ingress-nginx takes and NGINX Ingress Controller refused: the same
host in several rules of one Ingress (the paths of all of them), and an
Ingress with only spec.defaultBackend (a catch-all for every host, with
-allow-empty-ingress-host; with -ingress-nginx-defaults it serves plain
http as ingress-nginx does, where NGINX Ingress Controller sends it to https
because the default server has the default certificate).
Several Ingresses of one host¶
As ingress-nginx does, regular Ingresses that share a host make one server:
the oldest Ingress of the host holds it, and the paths the others of its
namespace have for the host become locations of its server. The annotations of
each Ingress apply to its own paths; the settings of the server (redirects,
HSTS, server snippets) are those of the holder, and its certificate is that of
the holder or, if it has no TLS for the host, of the first other Ingress that
has. A path the server has already stays with the Ingress that has it first,
with a warning on the other. An Ingress whose every host is merged so gets an
event that names the Ingress whose server has its paths; a change to it
changes that server. Masters and minions (nginx.org/mergeable-ingress-type)
and VirtualServers keep the controller's rule: the host is theirs, and the
other resources of the host are rejected.
Hosts across namespaces: -ingress-nginx-cross-namespace¶
ingress-nginx merges the Ingresses of a host whatever their namespace, and a
canary Ingress of any namespace attaches to the Ingresses of the host it
names. In a cluster shared by teams, that lets whoever may create an Ingress
in one namespace add paths to the host of another namespace - a /login in
front of someone else's application - or send its traffic, or a share of it,
to a Service of their own with a canary. wxingress therefore keeps a host in
the namespace of the Ingress that holds it unless told otherwise:
- an Ingress of another namespace with that host loses it, with the warning
host <host> is taken by an Ingress of another namespace(and, if it has no other host, the eventAll hosts are taken by other resources); the warning names neither the holder nor its namespace; - a canary Ingress attaches only to the Ingresses of its own namespace; one
that finds its host and path only in another namespace gets the event
The canary has no Ingress with the same host, path and path type in its namespace. With-ingress-nginx-defaultsa canary is the canary only of upstreams (namespace, Service, port) of its own namespace, and the upstream is told by namespace, Service and port, not by ingress-nginx's name<namespace>-<service>-<port>, which namespacea-bwith Servicecshares with namespaceawith Serviceb-c.
Clusters that rely on ingress-nginx's behaviour - a WAF or an authentication
proxy in a namespace of its own that adds a canary or paths to the hosts of
other namespaces - turn it on with -ingress-nginx-cross-namespace (Helm:
controller.ingressNginxCrossNamespace: true). It is not part of
controller.ingressNginxCompat: turn it on only where everyone who may create
Ingresses may also take over the hosts of every namespace.
Server aliases¶
A server-alias never takes a host away. nginx serves a name with the server
it reads first, so an alias that competed with a host would decide by the
order of the config files. As ingress-nginx skips an alias that is a host or
an alias already, wxingress leaves out of the server, with a warning on its
Ingress (server-alias <name> is ignored: ...), every alias that
- is the host of another resource (Ingress, VirtualServer, TLS-passthrough
TransportServer), compared without case, or, as a wildcard
(
*.example.com), matches one, or falls under another resource's wildcard host - nginx prefers the exact name and the longer wildcard, so either would take requests from the other; - collides in the same way with an alias of an older Ingress.
An alias that is a host of the Ingress itself is no conflict and simply not repeated; an alias is given once, to the first server of the Ingress. When the other resource goes, the alias comes back with the next sync.
Prefix paths¶
A path of type Prefix matches the path and what is below it, as the
Kubernetes API and ingress-nginx have it: /p matches /p and /p/..., not
/px. As in ingress-nginx it is rendered as location /p/ and location = /p
(without the second when the Ingress has an Exact path /p); / and paths
that end in / stay as they are, and so do paths that are regular expressions
(use-regex, nginx.org/path-regex) or are rewritten.
Reloads¶
With -dynamic-upstreams NGINX is reloaded, as ingress-nginx reloads it, only
when its configuration changes. The configuration does not change with the
servers of an upstream: ingress-nginx compares its configuration without the
endpoints of its backends, and here a config whose file differs only in the
server lines of its upstreams - addresses and their parameters, not a server
by name, which NGINX resolves when it loads the config - is not a change. The
file still gets those servers, since a reload starts the upstreams with what
the file has, and the servers go to the wxingress module through its upstream
API. An update of an Ingress or a VirtualServer that changes nothing else -
a label, an annotation that renders the same, new endpoints - reloads nothing:
the module gets the servers, canaries and certificates that changed. The
controller remembers which servers the module has for each upstream, so it
calls the API only for those that changed. A TransportServer is handled the
same way: the upstreams of the stream block that its config has get their
servers through the stream part of the same API (/api/9/stream/upstreams/),
with the max_fails, max_conns and fail_timeout its upstream gives them,
so new endpoints of a TCP or UDP service reload nothing either. The same
holds for the ConfigMap: a change of it regenerates every config, and reloads
only when one of them, the main config or the dhparam changed.
An EndpointSlice that is added, changed or deleted queues its Service, once however many of its EndpointSlices change - a deleted one names its Service in its labels, so its servers go too, also when its Service stays. The Services whose EndpointSlices changed get their servers together, at once after a quiet time and then at most ten times a second, and only the upstreams whose servers changed are sent. Their configs are not regenerated and written for every change: the file of a config gets its new servers right before the next reload, so a reload never starts an upstream with servers it no longer has.
Reloads are rate limited as ingress-nginx's syncs are: -sync-rate-limit
(Helm: controller.syncRateLimit) is the most reloads a second, by default
0.3 as in ingress-nginx. A change after a quiet time is reloaded at once; one
that comes sooner after a reload waits for the end of the interval, and every
change that comes meanwhile is in that one reload. What needs no reload -
servers, canaries, certificates, basic-auth files - does not wait, except a
canary rule while a reload waits, since it may name an upstream that reload
brings. When events come faster than the controller handles them, it handles
them as one batch and reloads at its end when something in it changed the
configuration, whatever the events were.
Certificates¶
With -dynamic-upstreams the certificates of hosts are not named in the
configuration: as in ingress-nginx, the controller sets them by name in the
wxingress module, in shared memory, and the module picks one for each TLS
handshake by the name the client sent - that name, in lower case and without a
trailing dot, then the wildcard one level above it (*.example.com for
a.example.com, not for a.b.example.com), then _, the default certificate.
A TLS Secret that is added, changed or removed, and an Ingress that names
another Secret, change no configuration and cause no reload: the next handshake
has the new certificate. Each worker parses a certificate once and keeps it
until the controller sets another for its name.
Every TLS server names the default server's certificate
(/etc/nginx/secrets/default), which is what a name without a certificate of
its own gets - also a host whose Secret is missing or invalid, which ingress-nginx
serves the default certificate as well, where the controller without the module
rejects the handshake. Without -default-server-tls-secret that is the fake
certificate of ingress-nginx ("Kubernetes Ingress Controller Fake Certificate",
self-signed for ingress.local), made when the controller starts.
Since a TLS Secret then changes no configuration, its events regenerate
nothing, as in ingress-nginx, where a Secret only ends up in the next sync: the
controller writes or removes the Secret's file and gives the module what
changed in one go, at most twice a second - however many Secrets came, changed
or went since the last time (ingress-nginx syncs at most every 3.3 s by
default, --sync-rate-limit=0.3). Only a Secret that becomes valid or invalid,
comes or goes has the Ingresses and VirtualServers that name it regenerated,
once for all such Secrets in that go, for the hosts the module is to have and
for their warnings; the configuration stays the same. A Secret that is also
anything else - a special Secret, in a Policy, a CA, a JWK, for basic auth, of
a TransportServer or a minion - is handled as before. On kind, rotating all
1000 Secrets of an Ingress with 1000 hosts was served 0.1 s after the API
server had them (more than a minute before), and deleting the namespace with
them took 16 s (5 min before).
The Secrets are still written to /etc/nginx/secrets, and what the module is
given is what is in those files. The module keeps its certificates over reloads
and starts without any, so the controller gives it all of them after the first
reload when NGINX starts, as ingress-nginx sends all of them after it starts.
The module's API, on the socket of the upstream API:
| Request | |
|---|---|
GET /api/wxingress/1/certificates |
every name, with the generation of its certificate |
GET /api/wxingress/1/certificates/:name |
one name, its certificate without the key |
PUT /api/wxingress/1/certificates/:name |
{"certificate": PEM, "key": PEM}; refused with 400 when they do not belong together |
DELETE /api/wxingress/1/certificates/:name |
A name is a host, *. and a host, or _. The module holds up to 32768 names
and 128 MB of PEM.
Annotations¶
| Annotations | Support |
|---|---|
proxy-connect-timeout, proxy-read-timeout, proxy-send-timeout |
Whole seconds, rendered with s. |
proxy-body-size, client-body-buffer-size |
client_max_body_size (k, m, g) and client_body_buffer_size (k, m). |
proxy-buffering, proxy-request-buffering |
on/off. |
proxy-buffer-size, proxy-buffers-number, proxy-busy-buffers-size |
Rendered as ingress-nginx does: proxy_buffer_size S and proxy_buffers N S, so either sets both (N defaults to 4, S to 4k or the ConfigMap's). |
proxy-max-temp-file-size, proxy-http-version |
As in ingress-nginx; proxy-http-version takes 1.0 or 1.1. |
proxy-next-upstream, proxy-next-upstream-timeout, proxy-next-upstream-tries |
With ingress-nginx's validator; non_idempotent is moved to the end. |
rewrite-target, use-regex, x-forwarded-prefix |
A rewrite-target or use-regex makes every path of the server (of every Ingress on the host) a case-insensitive regular expression ~* "^path", ordered longest first as ingress-nginx orders them; rewrite-target with $1-style captures. x-forwarded-prefix is sent only from the locations rewrite-target rewrites. nginx.org/rewrite-target and nginx.org/rewrites win over rewrite-target, nginx.org/path-regex over use-regex. |
permanent-redirect, permanent-redirect-code, temporal-redirect, temporal-redirect-code |
Every location of the Ingress answers return <code> "<url>", as in ingress-nginx: temporal-redirect wins with 302, permanent-redirect has 301, a code outside 300-308 is the default with a warning. The URL must be http or https; a $ may only start an nginx variable (https://example.org$request_uri). |
service-upstream |
Proxies to the ClusterIP of the Service; a headless Service to its endpoints. |
ssl-redirect, force-ssl-redirect, preserve-trailing-slash |
Plain http to https as ingress-nginx sends it: https://<Host header without its port><URI> with ingress-nginx's code (308 with -ingress-nginx-defaults) and the trailing slash of the URI dropped unless preserve-trailing-slash. ssl-redirect (default true) where the host has TLS, force-ssl-redirect also without. ACME challenges and the health URI are not redirected. As in ingress-nginx the redirect is sent by each location, in its rewrite phase: after nginx answered 413 to a body over the location's proxy-body-size, after the location's own returns (permanent-redirect, a CORS preflight, a location that answers 503), before rewrite-target's rewrite. Whether a host redirects is decided per server (the first Ingress of a host has its say), where ingress-nginx decides per location. |
enable-cors, cors-allow-origin, cors-allow-methods, cors-allow-headers, cors-allow-credentials, cors-expose-headers, cors-max-age |
ingress-nginx's CORS with its defaults and validators: the headers for an allowed origin (* without cors-allow-origin; https://*.example.com takes one label), a preflight (OPTIONS) of an allowed origin answered 204 before auth, other origins without headers and to the backend. add_header ... always with add_header_inherit merge instead of more_set_headers, so the server's headers stay. |
auth-tls-secret, auth-tls-verify-client, auth-tls-verify-depth, auth-tls-error-page, auth-tls-pass-certificate-to-upstream, auth-tls-match-cn |
Client certificates on the server of the host, from the ca.crt (and ca.crl) of the Secret, kept in step with it: ssl_verify_client (default on), ssl_verify_depth (1), error_page 495 496, the ssl-client-verify, ssl-client-subject-dn, ssl-client-issuer-dn headers (and ssl-client-cert) to the backend, auth-tls-match-cn as 403 client certificate unauthorized. A Secret of another namespace or a missing one make the locations answer 503. A host without TLS has https with the default certificate (-ingress-nginx-defaults, as in ingress-nginx), where the client certificate is checked; its plain http answers 503 (ingress-nginx serves it without the check). Without the defaults, a host without TLS makes the locations answer 503. An nginx.org IngressMTLS Policy wins. |
from-to-www-redirect |
A server for www.<host> (or <host> without www. for a www. host) that answers everything with ingress-nginx's redirect <scheme>://<host>:<port><URI without trailing slash>, with TLS where the Ingress's TLS covers that name - unless a resource has that name; when one takes it or lets it go, the Ingress is configured again. |
custom-http-errors, default-backend |
default-backend names a Service of the namespace (its first port), kept in step with its endpoints. custom-http-errors (300-599) sends those answers of the backends to it as ingress-nginx does: /, with X-Code, X-Format, X-Original-URI, X-Namespace, X-Ingress-Name, X-Service-Name, X-Request-ID. And a 502 or 503 goes to it with the request as it was - nginx's own when the Service has no endpoints (ingress-nginx's case), but also one a backend answers itself. Without default-backend the codes go to the controller's -default-backend-service (below), as in ingress-nginx, without the 502/503 fallback; without either custom-http-errors is ignored with a warning. |
mirror-target, mirror-host, mirror-request-body |
mirror to an internal location that proxies to the target (an http(s) URL, nginx variables allowed - $request_uri is the mirrored request's), resolved per request with the pod's name servers, Host the target's or mirror-host; the mirror's answer is not waited for. |
proxy-ssl-secret, proxy-ssl-verify, proxy-ssl-verify-depth, proxy-ssl-ciphers, proxy-ssl-protocols, proxy-ssl-name, proxy-ssl-server-name |
TLS to the backends (with backend-protocol: HTTPS) with ingress-nginx's defaults - verify off, depth 1, TLSv1.2, DEFAULT ciphers, server name off: the client certificate from tls.crt/tls.key and the CA from ca.crt of a Secret of the namespace (503 if it has neither). An nginx.org egressMTLS Policy wins. |
ssl-passthrough |
With -enable-tls-passthrough (chart controller.enableTLSPassthrough, ingress-nginx's --enable-ssl-passthrough): per host a TransportServer of the TLS passthrough listener, made by the controller, that sends the TLS connections of that name untouched to the Service of path / (or the first path), endpoints without a reload. Without the flag ignored with a warning, as ingress-nginx ignores it without its flag. |
enable-access-log |
false: access_log off in the locations of the Ingress. |
enable-rewrite-log |
true: rewrite_log on in the locations of the Ingress (the rewrites are logged in the error log at notice). |
proxy-cookie-domain, proxy-cookie-path |
proxy_cookie_domain and proxy_cookie_path in the locations of the Ingress, with the value <from> <to> or off; another value is ignored with a warning. |
relative-redirects |
true: absolute_redirect off in the locations of the Ingress, so nginx's own redirects are relative; false leaves nginx's default. |
use-port-in-redirects |
port_in_redirect on (true) or off (false) in the locations of the Ingress. |
satisfy |
satisfy any or satisfy all in the locations of the Ingress: with any, a request that passes one of the source ranges, basic auth and external auth is served. Another value is ignored with a warning. |
disable-proxy-intercept-errors |
true: the codes of custom-http-errors are not intercepted, the backends' answers go to the client as they are. A default-backend still gets the 502 and 503 (above). |
server-alias |
More names of the server, comma- or space-separated host names. A wildcard stands for one label and must leave at least two (*.example.com; *.com, * and example.* are ignored with a warning). |
upstream-vhost, app-root, ssl-ciphers, ssl-prefer-server-ciphers, proxy-redirect-from, proxy-redirect-to |
As nginx.org/upstream-vhost, nginx.org/app-root, nginx.org/ssl-ciphers, nginx.org/ssl-prefer-server-ciphers, nginx.org/proxy-redirect-from and -to (which win); a value with characters that would break out of the directive is ignored with a warning, as is one the controller would refuse in the nginx.org/* annotation (#, a backtick, a regular expression PCRE does not compile in proxy-redirect-from). proxy-redirect-from other than off or default without proxy-redirect-to, and proxy-redirect-to without proxy-redirect-from, are ignored with a warning. |
backend-protocol |
HTTPS, GRPC, GRPCS for every Service of the Ingress (as nginx.org/ssl-services, nginx.org/grpc-services); HTTP, AUTO_HTTP as they are; FCGI below; any other answers 503. |
backend-protocol: FCGI, fastcgi-index, fastcgi-params-configmap |
The locations pass to their upstreams with FastCGI as in ingress-nginx: nginx's fastcgi_params, fastcgi_index, and a fastcgi_param for each key of a ConfigMap of the Ingress's namespace (watched; keys and values validated as ingress-nginx does). A missing ConfigMap, one of another namespace or an invalid parameter: 503. |
allowlist-source-range (whitelist-source-range), denylist-source-range |
Comma-separated addresses and CIDRs, IPv4 and IPv6, in every location of the Ingress as ingress-nginx renders them: deny of each denied range first, then allow of each allowed one and deny all (403). allowlist-source-range wins over whitelist-source-range. An invalid range makes the locations answer 503, as ingress-nginx denies them. |
limit-rps, limit-rpm, limit-connections, limit-burst-multiplier, limit-allowlist (limit-whitelist), limit-rate, limit-rate-after |
ingress-nginx's zones (5m, keyed by $binary_remote_addr), allowlist (limit-allowlist wins over limit-whitelist), bursts (limit × multiplier, default 5, nodelay) and its 503; limit-rate and limit-rate-after in kilobytes (limit_rate <n>k). An invalid limit or allowlist drops the whole rate limit, as there. nginx.org/limit-req-rate wins over limit-rps and limit-rpm. |
auth-type, auth-secret, auth-secret-type, auth-realm |
Basic auth from the Opaque Secret (auth-file with key auth, or auth-map), kept in step with the Secret; the Authorization header is not passed on. auth-type: digest answers 503: open-source nginx has no digest auth. |
auth-url, auth-method, auth-signin, auth-signin-redirect-param, auth-response-headers, auth-request-redirect, auth-snippet, auth-cache-key, auth-cache-duration, auth-always-set-cookie |
External auth with auth_request, its internal location and headers as in ingress-nginx, and the 302 to the sign-in URL with rd= on 401. auth-snippet needs -enable-snippets. |
auth-proxy-set-headers |
The headers of a ConfigMap of the Ingress's namespace (name or <namespace>/name) to the auth service, nginx variables in the values as in ingress-nginx; watched, a change configures the Ingress again. A missing ConfigMap, one of another namespace (wxingress takes no resources of other namespaces), an invalid header name or a value ingress-nginx would not take as a header value (a line break or other control character) or with a variable not every server knows: 503. |
auth-keepalive, auth-keepalive-requests, auth-keepalive-timeout, auth-keepalive-share-vars |
The auth service as an upstream with keepalive (default 1000 requests, 60 s), its name resolved while nginx runs (server ... resolve), so a missing auth service fails no reload. As in ingress-nginx not with HTTP/2 on (the ingress-nginx default; ignored with a warning) and not when the host of auth-url has a variable. nginx keeps the connection of an auth answer without a body (oauth2-proxy's 202, a Content-Length: 0) and closes one with a body, which auth_request does not read. auth-keepalive-share-vars is what nginx's subrequests do anyway (they share the variables of the request). |
connection-proxy-header |
close or keep-alive: the Connection header to the backend instead of the one the location sends ($connection_upgrade, or none for HTTP/2) - exactly one. |
custom-headers |
The headers of a ConfigMap of the Ingress's namespace on every answer of its locations, also nginx's own, the backend's of the same name replaced (proxy_hide_header and add_header ... always, with add_header_inherit merge so the server's stay - where ingress-nginx has more_set_headers). As in ingress-nginx each header must be in the ConfigMap key global-allowed-response-headers (comma-separated), and a $ of a value is a $. Not allowed, missing, of another namespace, an invalid name or value: 503. The ConfigMap is watched (an informer for just it, while an Ingress names it); a change reloads. |
enable-opentelemetry, opentelemetry-trust-incoming-span, opentelemetry-operation-name |
Tracing of the locations with nginx's otel module: otel_trace on or off, otel_trace_context propagate (trusting the span of the request, the default) or inject, otel_span_name. The traces go to the collector of the ConfigMap's otel-exporter-endpoint (otlp-collector-host/-port) and nowhere else; without one the annotations are ignored with a warning. |
stream-snippet |
With -enable-snippets: a file of the stream block per Ingress (stream-conf.d/ingress-snippets_<namespace>-<name>.conf), gone with the Ingress. |
server-snippet, configuration-snippet |
The server and location snippets, as nginx.org/server-snippets and nginx.org/location-snippets, which win over them. They need -enable-snippets; without it they are ignored with a warning. |
affinity: cookie, affinity-mode, affinity-canary-behavior, session-cookie-name (default INGRESSCOOKIE), session-cookie-path, session-cookie-domain, session-cookie-expires, session-cookie-max-age, session-cookie-secure, session-cookie-samesite, session-cookie-conditional-samesite-none, session-cookie-change-on-failure |
With -dynamic-upstreams, ingress-nginx's sticky balancers in the wxingress module (see below): affinity-mode balanced or persistent, session-cookie-change-on-failure, and its cookie, <key>\|<backend key> with Expires, Max-Age, Domain, Path, Secure, HttpOnly and SameSite as ingress-nginx writes it, so its sessions stay pinned. Without it nginx's sticky cookie, HttpOnly, with the path of the Ingress, and session-cookie-change-on-failure always true. An nginx.org/sticky-cookie-services cookie of the same Service wins. |
upstream-hash-by, upstream-hash-by-subset, upstream-hash-by-subset-size, load-balance |
load-balance is round_robin or ewma, the subset size 3 by default; with affinity both are ignored, and nginx.org/lb-method wins over them. With -dynamic-upstreams, ingress-nginx's chash, chashsubset, round_robin and ewma in the wxingress module (see below). Without it hash ... consistent; load-balance: ewma becomes least_time last_byte, round_robin nginx's round robin, and upstream-hash-by-subset hashes to one endpoint, with a warning. |
canary, canary-weight, canary-weight-total, canary-by-header, canary-by-header-value, canary-by-header-pattern, canary-by-cookie |
Need -dynamic-upstreams (the wxingress data plane); without it, or with an invalid canary annotation, the canary Ingress is ignored with an event. canary-weight-total defaults to 100. A canary Ingress makes no servers of its own. With -ingress-nginx-defaults, as ingress-nginx's alternative backend: a path of the canary finds the location with its host, path and path type (in the canary's namespace, in any namespace with -ingress-nginx-cross-namespace), and the canary's backend becomes the canary of that location's upstream - ingress-nginx's backend of namespace, Service and port - so a request with the canary's header or cookie, or its weight, goes to the canary on every host and path of every Ingress whose backend is that Service's port; an upstream has the first such canary, oldest first, as ingress-nginx uses only its first alternative backend. Without -ingress-nginx-defaults the canary is that of the location's Ingress alone. Decided per request; a changed rule (weight, header, cookie) reaches the module without a reload, a canary that attaches to an upstream or leaves one reloads the configs of its Ingresses (ingress-nginx's needs none: its locations name the upstream at request time, wxingress's have the canary upstream in their config). |
TCP and UDP services, the default backend¶
-tcp-services-configmap and -udp-services-configmap (chart
controller.tcpServices and controller.udpServices, which the chart puts
into a ConfigMap, opens in the pods - with controller.hostPort.enable on the
nodes too - and adds to its Service; or controller.tcpServicesConfigMap for
an existing one) take ingress-nginx's format, "<port>":
"<namespace>/<service>:<port>[:PROXY][:PROXY]": the port of the Service by
number or name, the first PROXY taking the PROXY protocol of the client
(listen ... proxy_protocol), the second sending it to the Service
(proxy_protocol on; <port>::PROXY for that alone), not for UDP. Each entry
is a stream server with ingress-nginx's settings - proxy_timeout 600s, the
next upstream on, 3 tries in 600 s, one response a UDP datagram - made by the
controller (no TransportServer object), whose endpoints change without a
reload with -dynamic-upstreams; the ConfigMap is watched, an entry that is
added or taken out reloads. An entry the controller cannot take - not a port,
a Service or port that does not exist - is a Warning event on the ConfigMap.
-default-backend-service=<namespace>/<name> (chart
controller.defaultBackendService) is ingress-nginx's
--default-backend-service: its first port answers the requests of an
unknown host (in place of the default 404; a default-server-return of the
ConfigMap wins), the paths of a host that no Ingress of it has (a server
without / gets a location / to it), and the custom-http-errors of
Ingresses without a default-backend annotation. Its endpoints change without
a reload. An Ingress with only spec.defaultBackend (a catch-all) takes the
unknown hosts instead, as in ingress-nginx. A Service without endpoints is no
default backend for nginx's own 503s, as in ingress-nginx.
Metrics¶
With -enable-prometheus-metrics (chart prometheus.create, on the port of
prometheus.port - 10254 is ingress-nginx's) the controller exports
ingress-nginx's metrics beside its own, under ingress-nginx's names, labels
and buckets, so that its dashboards (Grafana 9614) and alerts work as they
are. Their constant labels are controller_namespace, controller_pod and
controller_class - the IngressClass controller served in ingress-nginx's
place, k8s.io/ingress-nginx where that is one of
-ingress-class-controllers.
| metric | as ingress-nginx |
|---|---|
nginx_ingress_controller_requests |
a counter per status, method, path (the Ingress's), namespace, ingress, service, canary (the upstream a canary sent the request to, else empty) and host |
..._request_duration_seconds, _response_duration_seconds, _connect_duration_seconds, _header_duration_seconds |
histograms of $request_time and of the upstream's response, connect and header times (only for a request that tried one upstream, as ingress-nginx's Lua cannot read a list), prometheus.DefBuckets |
..._ingress_upstream_latency_seconds |
the summary ingress-nginx exported up to 1.11 (removed in 1.12, so 1.12's and 1.15's dashboards do not use it): $upstream_connect_time of a request that tried one upstream, per ingress, namespace, service and canary, with the quantiles 0.5, 0.9 and 0.99 of the last 8 to 10 minutes, as client_golang's summary with ingress-nginx's objectives and its default age buckets. The module counts the times in bins - a millisecond each up to 63 ms (nginx's times have milliseconds), then 16 to an octave up to 131 s - and the controller takes the quantiles from the difference of two readings some ten minutes apart, a reading kept every two minutes of the scrapes: exact to the millisecond up to 63 ms, within a sixteenth above, where ingress-nginx's is a sample within the objectives' rank error |
..._request_size, _response_size, _bytes_sent |
histograms of $request_length and $bytes_sent, LinearBuckets(10, 10, 10) and ExponentialBuckets(10, 10, 7) |
..._config_hash, _config_last_reload_successful, _config_last_reload_successful_timestamp_seconds, _success, _errors |
the reloads; the hash is a number that changes with every configuration loaded, not a hash of the file |
..._ssl_expire_time_seconds, _ssl_certificate_info |
per host of an Ingress or VirtualServer with TLS, labels namespace (of the Secret), class, host, secret_name, identifier and the issuer, serial and key algorithm; a certificate file is parsed again when it changed |
..._nginx_process_* |
the nginx processes, master and workers: connections and requests from stub_status, CPU, memory, I/O, their number and the oldest start time from /proc |
The request metrics need -dynamic-upstreams: the wxingress module counts
them in nginx's log phase (dataplane/src/metrics.c), for the locations the
controller's config marks with $wxingress_metrics - the labels of the
location in one constant string, so the module reads one variable, builds
nothing and allocates nothing per request. The counts are in shared memory
that outlives reloads, kept per worker and summed when the controller reads
them (GET /api/wxingress/1/metrics on the module's socket) at each scrape.
The table has 8192 series; a request beyond them is counted in the series
without labels. ingress_upstream_latency_seconds has a table of its own of
2048 series (namespace, Ingress, Service, canary), found through the same
per-worker cache as the location's series, and costs three atomic adds a
request that connected to an upstream. Measured on a trivial proxy (one worker and four, perf
stat, instructions per request): +2.0 % (about 630 of 31 200), of which the
module's step and the extra set are some 300 by the profile and the rest
is spread over nginx; on a request with TLS and real headers it is less.
With the admission webhook (below), ..._check_success and
..._check_errors count its checks per namespace and ingress, as
ingress-nginx's do.
Not exported: orphan_ingress, leader_election_status, build_info,
the admission webhook's admission_* gauges of the last check's size and
durations, and the request metrics of VirtualServers and Gateway API
routes, which have none in ingress-nginx.
The admission webhook¶
With -validating-webhook=:8443 (chart controller.admissionWebhooks,
ingress-nginx's values: enabled, failurePolicy - Fail by default, as
ingress-nginx's - timeoutSeconds, port, namespaceSelector,
objectSelector, service, the self-signed certificate of the
patch jobs with kube-webhook-certgen or certManager) the controller
serves a ValidatingWebhook for the Ingresses created and updated
(/networking/v1/ingresses, AdmissionReview v1). As ingress-nginx's
CheckIngress it lets through an Ingress of another class or one being
deleted, and refuses:
- a snippet annotation (
*-snippet,*-snippets) while snippets are disabled (-enable-snippets, ingress-nginx'sallow-snippet-annotations), with ingress-nginx's message; - an Ingress the controller's validation (
internal/k8s/validation.go) rejects - which without the webhook is not served, with a Warning event; - a path of a host that another Ingress of the class has, whatever their
path types, unless one of the two has the
canaryannotation, with ingress-nginx's message (host "<host>" and path "<path>" is already defined in ingress <namespace>/<name>) when that Ingress is of the same namespace; for one of another namespace the message is... is already defined in another Ingress, which tells whoever creates the Ingress neither the namespace nor the name of someone else's; - an Ingress whose configuration does not render or does not pass
nginx -t: the configuration is built aside as it would be with the Ingress - its own config, those of the Ingresses of its hosts it would be merged into, those of the upstreams it would be the canary of - and tested with the files nginx runs with but those replaced (-validating-webhook-nginx-test, on by default). ingress-nginx stopped this test for CVE-2025-1974: whoever can reach the webhook can have its configuration parsed, and a snippet runs innginx -t. wxingress tests only while snippets are disabled, when no annotation can bring a directive of its own.
The warnings of the Ingress's config (an annotation that is ignored, a host
another resource has) go back with the answer, which kubectl prints. A check
waits for the controller's sync to finish what it is doing, and the sync for
the check while it builds and renders the configs: they read the same state.
nginx -t runs afterwards, without holding the sync up, on a copy of the
configuration taken while the check held it, one check at a time, and is
stopped after 10 seconds (the API server's default timeout of a webhook). A
Secret the check needs for nginx -t that no resource uses yet is written for
the test only and removed when the check is done. The webhook takes at most 4
reviews at once - more are answered 503, a failed call, so the webhook's
failurePolicy decides - and none bigger than 3.5 MiB (413). The
certificate is read again when its files change (a renewed Secret).
The webhook does not authenticate its callers: API servers present no client
certificate by default, and whoever reaches the port can have Ingresses
checked (built, rendered and, with -validating-webhook-nginx-test, parsed by
nginx -t). Restrict who reaches it with a NetworkPolicy that admits only the
API server to the webhook port. the RKE2 runbook, step 5, puts
it in the place of rke2-ingress-nginx's; dataplane/e2e/admission-webhook.sh
checks what it admits, refuses and counts.
The Gateway API: the same annotations on routes¶
With -enable-gateway-api, an HTTPRoute or GRPCRoute takes the ingress-nginx
annotations of the backends' upstreams - proxy-*-timeout,
proxy-body-size, client-body-buffer-size, proxy-buffering,
proxy-buffer-size, proxy-buffers-number, proxy-busy-buffers-size,
proxy-next-upstream*, proxy-http-version, service-upstream,
load-balance, upstream-hash-by*, affinity, affinity-mode,
session-cookie-*, limit-rps, limit-rpm, limit-connections,
limit-burst-multiplier, limit-allowlist, limit-rate, limit-rate-after,
canary-by-header, -header-value, -header-pattern, canary-by-cookie,
backend-protocol (HTTPS and GRPCS turn TLS to the backends on; an
egressMTLS Policy adds the client certificate and verification), and
with -enable-snippets server-snippet, configuration-snippet,
nginx.org/server-snippets and nginx.org/location-snippets - read by the
same code as on an Ingress; with -dynamic-upstreams the balancer and
affinity are the module's, as below. The canary annotations apply to a rule
with two backends: the second is the canary, the weights are the backends'
(the second may have weight 0, for a canary only the header or cookie
reaches).
Rate limits, basic and external auth, access control and the other
policies are NGINX Ingress Controller's Policies, attached with
wxingress.wizardtales.com/policies: rate, auth on the route or the
Gateway. A rule with two weighted backends is a canary of the module: its
weights change without a reload. Redirects, rewrites, header changes and CORS
are the Gateway API's own filters.
limit-whitelist is read as limit-allowlist. proxy-request-buffering,
proxy-max-temp-file-size and affinity-canary-behavior are accepted on a
route but not applied. The annotations that restrict who gets an answer -
auth-*, the source-range annotations, satisfy, ModSecurity and
global-rate-limit - are not taken on routes: rather than serve a route
without the restriction, its rules answer 503, with a warning. Use the
equivalent Policy instead.
Balancing without a reload: -dynamic-upstreams¶
With -dynamic-upstreams every upstream of an Ingress says meta_balancer;
after its method, and the wxingress module chooses the server of each request
and of each retry by a rule the controller sets through its API
(/api/wxingress/1/balancers/<upstream>), the way ingress-nginx's Lua balancer
did. The configuration is the same whatever load-balance, upstream-hash-by
and the affinity annotations say, so changing them reloads nothing; an
upstream without a rule keeps the method of its block (lb-method).
The algorithm is the one ingress-nginx's get_implementation picks - the
affinity, else upstream-hash-by (with upstream-hash-by-subset, chashsubset),
else load-balance - and each is the Lua one's, step for step:
| algorithm | as in ingress-nginx |
|---|---|
round_robin |
each worker in turn from a random start; a server the request already tried is passed over |
ewma |
two random servers of those the request has not tried, the lower Peak-EWMA score (connect + response time, 10 s decay). The averages are per address in shared memory: every worker uses them, and they survive reloads. A server added starts at the average of the others, one removed loses its average. |
chash |
resty.chash's ring (160 points a server, CRC32), keyed by upstream-hash-by with its nginx variables expanded; a key keeps its server, and a new server takes about 1/n of the keys |
chashsubset |
the ring over sets of upstream-hash-by-subset-size servers (3), a random server of the key's set |
sticky_balanced |
the cookie's key on the ring of servers; a new session gets a random key |
sticky_persistent |
the cookie's key is md5(backend + address); a new session goes to a random server, and stays there when the backend scales |
Sessions pinned by ingress-nginx stay pinned: the backend in the cookie is
md5(md5(name) + name) of ingress-nginx's backend name,
<namespace>-<service>-<port>, and the ring and the persistent keys are
computed as ingress-nginx computes them. A canary's sessions stay on the side
whose backend their cookie names, before header, cookie and weight.
Where it differs: servers are sorted by address before they are split into subsets, so every replica makes the same subsets; ewma averages a try in only when it did not fail, where ingress-nginx averages in the last try even when it failed (a refused connection takes no time and would look fast); a sticky balancer with no server left to try leaves the try to the block's method.
Defaults: -ingress-nginx-defaults¶
Without an annotation or ConfigMap key, wxingress uses the defaults of the
controller it is forked from, which differ from those of ingress-nginx. The
flag -ingress-nginx-defaults (Helm: controller.ingressNginxDefaults: true)
switches to the defaults of ingress-nginx (its config.go and nginx.tmpl)
for every setting the rendering of wxingress covers. Keys of the ConfigMap and
annotations still win over them. The values that are not specific to Ingresses
(timeouts, buffers, HSTS, TLS, worker settings) apply to VirtualServers too.
The ConfigMap keys are those of wxingress (proxy-connect-timeout,
proxy-buffering, keepalive, hsts-*, ...); with the switch the hsts-*
keys apply without hsts, hsts-preload is read, and the ConfigMap takes
ingress-nginx's keys as well (internal/configs/ingress_nginx_configmap.go): a
key of the same name as it is, a renamed one under the name of wxingress
(proxy-body-size as client-max-body-size, keep-alive,
keep-alive-requests, upstream-keepalive-connections,
max-worker-connections, max-worker-open-files, use-proxy-protocol,
enable-real-ip with forwarded-for-header and proxy-real-ip-cidr,
log-format-upstream, log-format-stream, log-format-escape-json, the
*-snippet keys, otlp-collector-host/-port, use-http2, hide-headers,
server-name-hash-*) - a key of wxingress itself wins - and
whitelist-source-range, allowlist-source-range, denylist-source-range
and block-cidrs as allow/deny in every server of the resources (an Ingress
with source ranges of its own has those, as in ingress-nginx; snippets do
not replace them). Keys wxingress needs nothing for
(strict-validate-path-type, allow-snippet-annotations, ...) are taken
silently. So are compute-full-forwarded-for, generate-request-id and
ssl-session-tickets, which are not read at all: a value other than the
default is dropped without a warning - compute-full-forwarded-for: "true"
still sends X-Forwarded-For $remote_addr, generate-request-id: "false"
still sends X-Request-ID, ssl-session-tickets: "false" leaves nginx's
tickets on. The wxingress keys limit-req-status-code, max-conns and
slow-start are taken without a warning too but are never parsed, so they
have no effect. Every other key is a Warning event on the ConfigMap, and those that
restrict access (global-auth-*, enable-modsecurity, block-user-agents,
...) also an error in the log that the Ingresses are served without them.
hack/ingress-nginx-values translates the controller.config of the chart the
same way and lists what it cannot take. A proxy-buffer-size without proxy-buffers sizes four buffers of that
size, as ingress-nginx does.
Taken from ingress-nginx with the switch¶
The first column names the setting by its ingress-nginx key. Not every one of
those keys is read from the ConfigMap: proxy-buffers-number,
proxy-request-buffering, proxy-next-upstream, proxy-next-upstream-timeout,
proxy-next-upstream-tries, proxy-redirect-from and, below,
proxy-http-version, proxy-cookie-domain, proxy-cookie-path and
limit-conn-status-code are refused as ConfigMap keys (a Warning event, the
key is ignored); for them only the default of the table and the annotation of
the same name per Ingress apply (for the buffers the wxingress key
proxy-buffers, "4 4k", sets the same). limit-req-status-code,
generate-request-id and compute-full-forwarded-for are taken without a
warning but have no effect: only the default of the table applies (see above).
| Setting (ingress-nginx key) | ingress-nginx | without the switch | with the switch |
|---|---|---|---|
proxy-connect-timeout |
5s | 60s | 5s |
client-body-buffer-size |
8k | not set (nginx: two pages, 8k on x86-64) | 8k |
proxy-buffering |
off | on | off |
proxy-buffer-size |
4k | not set (nginx: one page) | 4k |
proxy-buffers-number (annotation only) |
4 (proxy_buffers 4 4k) |
not set (nginx: 8 4k) |
4 4k |
proxy-request-buffering (annotation only) |
on | not set (nginx: on) | on |
proxy-next-upstream (annotation only) |
error timeout | not set (nginx: error timeout) | error timeout |
proxy-next-upstream-timeout (annotation only) |
0 | not set (nginx: 0) | 0 |
proxy-next-upstream-tries (annotation only) |
3 | not set (nginx: 0, every server) | 3 |
proxy-redirect-from (annotation only) |
off | not set (nginx: proxy_redirect default) |
off |
upstream-keepalive-connections |
320 | 0, no keepalive |
64 per Service port (see below) |
http-redirect-code |
308 | 301 | 308 |
hsts |
true | false | true |
hsts-include-subdomains |
true | false | true |
hsts-max-age |
31536000 | 2592000 | 31536000 |
hsts-preload |
false | always preload |
false |
use-http2 |
true | false | true |
server-tokens |
false | on | off |
ssl-protocols |
TLSv1.2 TLSv1.3 | not set (nginx: the same) | TLSv1.2 TLSv1.3 |
ssl-ciphers |
ECDHE/DHE AES-GCM and CHACHA20 list, ssl_prefer_server_ciphers on |
not set (nginx: HIGH:!aNULL:!MD5) |
ingress-nginx's list, prefer on |
max-worker-connections |
16384 | 1024 | 16384 |
worker-shutdown-timeout |
240s | not set | 240s |
variables-hash-max-size |
2048 | 1024 | 2048 |
limit-req-status-code (no effect as a key) |
503 | 429 for nginx.org/limit-req-* (ingress-nginx's rate limits answer 503 anyway) |
503 |
generate-request-id (no effect as a key) |
true: X-Request-ID to the upstream, the client's or $request_id |
not sent | sent |
compute-full-forwarded-for (no effect as a key) |
false: X-Forwarded-For $remote_addr |
$proxy_add_x_forwarded_for |
$remote_addr |
| WebSocket (not a setting: always in ingress-nginx) | Upgrade and Connection in every location |
only for nginx.org/websocket-services |
every Ingress location |
| Other headers to the upstream | X-Scheme, X-Forwarded-Scheme, X-Original-Forwarded-For, Proxy "" |
not sent | sent |
| The same path twice for a host of one Ingress (not a setting: a host listed twice, a path listed twice) | the first one's location | two location of the path, a configuration nginx refuses (duplicate location) for every resource |
the first one, as ingress-nginx |
| The port of an Ingress backend (not a setting) | a port of the Service by its number or name, or else by its targetPort (a backend's 5000 is the Service's port 80 to 5000) |
the port's number or name only: "no port 5000 in service", 502 | also the targetPort, a port with that number or name first |
upstream-keepalive-connections: in ingress-nginx the 320 are the idle
connections one worker keeps for all backends together, in the single
upstream of its Lua balancer. In wxingress every Service port is an upstream
of its own with its own keepalive, and nginx allocates each one's cache
whole when it reads the configuration - some 140 bytes an entry, 45 kB an
upstream at 320, 5 MB of every pod for a hundred Ingresses. So the default is
64 idle connections per worker and Service port: a Service that has more
than 64 requests in flight on one worker opens a connection for each beyond
them, where ingress-nginx would keep up to 320 open. Setting the key in the
ConfigMap gives every upstream that number.
The same either way¶
| Setting | ingress-nginx | wxingress |
|---|---|---|
proxy-read-timeout, proxy-send-timeout |
60s | 60s |
proxy-body-size |
1m | 1m |
proxy-busy-buffers-size |
not set | not set |
proxy-max-temp-file-size |
1024m | not set (nginx: 1024m) |
proxy-http-version (annotation only) |
1.1 | not set (nginx 1.31: 1.1); left unset so gRPC and appProtocol: kubernetes.io/h2c keep their own |
proxy-cookie-domain, proxy-cookie-path (annotations only) |
off | not set (nginx: off) |
ssl-redirect |
true | true |
keep-alive, keep-alive-requests |
75s, 1000 | 75s, 1000 |
worker-processes |
number of CPUs | auto |
variables-hash-bucket-size |
256 | 256 |
server-name-hash-max-size |
1024 | 1024 |
error-log-level |
notice | notice |
use-forwarded-headers |
false | false |
ssl-early-data, ssl-ecdh-curve |
off, auto | not set (nginx: off, auto) |
client-header-buffer-size, large-client-header-buffers, client-header-timeout, client-body-timeout |
1k, 4 8k, 60s, 60s | not set (nginx: the same) |
http2-max-concurrent-streams |
128 | not set (nginx: 128) |
enable-underscores-in-headers, ignore-invalid-headers |
off, on | not set (nginx: the same) |
limit-conn-status-code (not a key) |
503 | 503 for ingress-nginx's limit-connections |
| default backend | 404 (--default-backend-service) |
404 (-default-backend-service) |
Not covered¶
These defaults of ingress-nginx differ from what wxingress renders and the switch does not change them, because wxingress does not render the directive or the difference is one of capacity, not of behaviour:
| Setting | ingress-nginx | wxingress |
|---|---|---|
ssl-session-cache, ssl-session-cache-size, ssl-session-timeout |
shared:SSL:10m, 10m |
nginx's: no shared cache, 5m |
ssl-session-tickets |
off | nginx's: on; the key is taken without a warning and has no effect |
ssl-buffer-size |
4k | nginx's: 16k |
upstream-keepalive-time, -timeout, -requests |
1h, 60s, 10000 | nginx's: 1h, 60s, 1000 |
map-hash-bucket-size |
64 | 256 (larger, only capacity) |
server-names-hash-bucket-size |
computed from the longest name | 256 |
log-format-upstream |
its upstreaminfo format | wxingress's main format |
default-type |
text/html | application/octet-stream |
use-gzip, enable-brotli |
off | off |
reuse-port, aio threads, tcp_nopush, reset_timedout_connection, log_subrequest, port_in_redirect off, types_hash_max_size 2048 |
on | nginx's |
X-Forwarded-Host, Host |
$http_host (with the client's port) |
$host |
X-Original-Forwarded-Host |
the client's X-Forwarded-Host |
not sent |
| SSL redirect | https://host and the URI without its trailing slash |
https://host:443 and the URI as it is |
| headers of gRPC locations | the same set as for HTTP | wxingress's gRPC headers |
proxy-stream-* (TCP/UDP services) |
600s, 3 tries | the same for -tcp-services-configmap; TransportServers have their own |
| ConfigMap keys of ingress-nginx | read | with the switch translated into those of wxingress (above) |