Moving from ingress-nginx to HAProxy

At my job we run over 100 customer WordPress sites on a single GKE cluster. Every request to those sites goes through a Kubernetes ingress controller. When this was built 8 or 9 years ago, ingress-nginx was the right choice. With the HollowByte vulnerabilities in OpenSSL and ingress-nginx being retired upstream, that's no longer the case, so we moved over to the HAProxy Kubernetes Ingress Controller.

Our second need was getting real client IPs all the way to WordPress. Bots are now the most common problem I deal with. Crawlers walk every translated page and run up our translation bill, scripts guess passwords against wp-login.php, and rendering crawlers knock pods over. You can't rate limit or block traffic you can't identify, and for a long time every request looked like it came from one of the 4 IPs inside the cluster.

So the plan was to fix client IPs at the load balancer first, then move to HAProxy and get the client IP right from the ingress to the pod as part of that move.

Why HAProxy

We went with HAProxy's community controller because it gets weekly security fixes, reads standard Kubernetes Ingress objects and has stick tables, which are perfect for the per-IP rate limiting I'm building next.

Getting the Real Client IP

The client IP has to survive two hops, and it can get lost at either one.

The first hop is load balancer to ingress which we tackled a week before the HAProxy work. We moved to a backend-service network load balancer with externalTrafficPolicy: Local, which keeps the client's source IP when traffic reaches the node. We kept the same reserved IP so DNS for 100+ sites didn't have to change. It also meant the HAProxy cutover later on only had to change which pods the load balancer's Service pointed at.

The second hop is ingress to WordPress. HAProxy deletes any X-Forwarded-For or Forwarded header the client sent and sets its own single value. Then a small mu-plugin in WordPress trusts that header only when the request comes from inside the cluster, and takes the rightmost entry. Once that was in place we could pass the real IP to tools like Gravity Forms for use with its Stripe rate limiter.¹

The important part is that the header is single-valued and set by the server. If a client can append its own value then anything downstream that trusts the header can be spoofed.

Translating the Config

Most of the work was moving our nginx config over to HAProxy without losing anything along the way. Body size limits, timeouts, HSTS, the HTTP to HTTPS redirect and our global IP blocklist all have direct equivalents in HAProxy so those were easy. The rest is where I spent my time.

Ingress Classes

The first thing I ran into was that the 3.2 controller only reads spec.ingressClassName to decide which Ingresses belong to it. Almost all of ours were old enough that they still used the deprecated kubernetes.io/ingress.class: nginx annotation, which HAProxy ignores, so as far as it was concerned they had no class at all. I didn't want to edit 100+ Ingresses in the middle of a cutover, and luckily the --empty-ingress-class flag tells HAProxy to serve those anyway. I cleaned up the annotations once nginx was gone.

Paths, Ports and QUIC

nginx matches on $uri, which has already been decoded and has any repeated slashes merged. HAProxy's path is the raw path the client sent. That sounds minor until you realize //wp-login.php or a percent-encoded . would walk right past a simple prefix match, so I used a regex ACL on the URL-decoded path instead. HAProxy 3.2 does have normalize-uri, but it's still marked experimental and I like to rely on tried-and-true systems, especially when something is new in our infrastructure. I'll look back at normalize-uri once we've been on HAProxy for a while.

The redirect port one would have been embarrassing. Inside the pod HAProxy listens on 8080 and 8443, and the controller's HTTPS redirect defaults to whatever port HAProxy is listening on. Without setting ssl-redirect-port: "443" every visitor who typed http:// would have been sent to :8443 and gotten nothing. I didn't spot this one by reading docs, the parity diff I talk about below caught it.

HAProxy also turns on HTTP/3 by default and tells browsers about it with an alt-svc header. Our network load balancer doesn't forward UDP on 443, so there was no point telling browsers to try it and I turned it off with --disable-quic.

Access Logs

This is the one that worried me the most because nothing would have told me it broke. Our per-site traffic alerts and the dry run for bot banning both read the ingress logs with regexes that were written for nginx's log format. If HAProxy logged in its own format the alerts wouldn't error out, they'd just never fire again. So I wrote a log-format that matches nginx field for field, including the [prod-<site>-80] token the alerts key on, and made sure the alerts still fired against it before we switched anything over.

Request Buffering

nginx reads the whole upload before it hands the request to Apache, HAProxy streams it straight through. With Apache prefork that means a slow upload ties up a PHP worker for the entire time it takes to upload. When I tested it, a 20 MB upload at 50 KB/s held one worker for about seven minutes, which isn't great when you only have so many workers per pod.

I couldn't make this go away completely. option http-buffer-request buffers small POSTs like logins and form submissions, and Apache's mod_reqtimeout drops clients that stall out completely.²

A Security Review Changed the Login Rules

Our nginx rules let Cloudflare's IP ranges skip the wp-login.php limits. Some of our sites sit behind Cloudflare, so all of their traffic arrives from Cloudflare IPs, and without the exemption we'd be rate limiting Cloudflare instead of visitors.

When I ran a security review on the HAProxy translation it pointed out that Cloudflare Workers can open raw TCP connections from those same IP ranges. A Worker can send any Host header it wants, so anyone could get the exemption on any of our sites, and only allowing it on the sites behind Cloudflare wouldn't fix that.

So the exemption is gone. Almost every login on our platform goes through single sign-on, and password logins are refused inside WordPress by a mu-plugin we shipped before this work, so nothing we rely on needed it.

Cloudflare traffic still needs some special handling though. We keep a list of the sites that are behind Cloudflare, and only on those sites do requests from Cloudflare's ranges get rate limited by CF-Connecting-IP, so visitors don't share one counter per Cloudflare edge server. Those counts live in their own table, with a per-edge-IP backstop so a forged header can't get around the limit. Cloudflare traffic on any other site is treated like everyone else, since a site that isn't behind Cloudflare has no reason to get traffic from it.

I'd rather enforce login rules in WordPress than at the proxy anyway. That way they stay correct no matter what ends up sitting in front of the site.

Taking Our Time with the Cutover

We're moving the whole layer that sends traffic to all our customers while the sites are live. I wanted to take things slow, let changes soak, and make sure we didn't interfere with a few new customer launches that happened during the change.

I started by running HAProxy in prod right beside nginx, behind a separate canary load balancer that only accepted my IP. Nothing customer-facing touched it, but I could hit any of our live sites through HAProxy and I used that setup for a while to see how it handled real sites.

Then I wrote a script to compare the two. It requested every host we serve, 529 of them plus a made-up host to make sure the default backend behaved, through both controllers and compared the status codes, redirect targets, headers and certificate fingerprints. The first run had 855 lines that didn't match. Most were harmless differences in headers, but I went through every one of them and fixed the real problems, the biggest being that :8443 redirect.

Reading pages only tells you so much, so I also went through the write paths on our test site through the canary: logging in, admin AJAX, uploading media and submitting a Gravity Forms form.

I also issued a real Let's Encrypt certificate through HAProxy before cutover. If HTTP-01 challenges are broken you don't find out right away, your certificates just start expiring a month later and then you find out all at once.

For TLS I ran testssl.sh against three sites through both controllers and got the same protocols and ciphers on each. A load test showed the same median response time, but HAProxy's p99 on static files was about 240ms where nginx was taking 2-3 seconds.

The cutover itself was about as small as I could make it. The HAProxy pods were running on the same five nodes as nginx, so the load balancer's health checks and backends didn't need to change. All we changed was which pods the Service selected.

During the cutover I had a probe hitting eight sites once a second. It recorded 0 failures across 4,425 requests after the switch. I left nginx running at full size for three days so I could roll back instantly if issues came up. None did, but it's better to be safe than to have a bunch of customer sites go down with no plan to bring them back up instantly.

What We Got

Here's the first overnight soak compared to the same window a week earlier on nginx.

The controller uses about twice the CPU, but on our nodes that's not something I'm going to notice. The WordPress pods behind it actually got a little quieter, with fewer pods and fewer errors.

The numbers aren't really why we did this though. Security patches come from upstream again so I'm not maintaining my own controller image, and we can finally see real client IPs and start working towards banning the people abusing our servers. That paid off quickly. Three days after cutover the traffic alerts caught a crawler hitting one site with 41,000 requests in 85 seconds, and because I could see exactly where it came from I blocked it at the ingress for every site.

It's not all better. We lost request buffering, which I covered above. HAProxy also picks the TLS certificate by SNI out of every certificate it knows about instead of going Ingress by Ingress, so I leaned on the fingerprint check in the parity diff to confirm it was serving the right certificate for each site. The controller also writes its own housekeeping messages into the same stream as the access logs, which means anything that parses those logs has to skip the lines that aren't requests.

Cleanup and One Last Surprise

Once nginx was scaled to zero, every Ingress got ingressClassName: haproxy and the dead nginx annotations came off. We deleted a handful of routes to services that hadn't existed for years, and found some old reserved IPs and expired load balancer certificates we were still paying for.³

The last step was dropping --empty-ingress-class. Two sites still depended on it for certificate renewals. The cert-manager version we were running created a separate challenge Ingress for HTTP-01 that only had the old nginx class annotation, and HAProxy ignores that. Every other site used http01-edit-in-place, where cert-manager adds the challenge path to the site's own Ingress instead. So I added that annotation to the two sites, dropped the flag and forced a renewal to make sure it worked.

Our first test didn't work. Turns out, on that cert-manager version, adding http01-edit-in-place to an existing Ingress doesn't update the Certificate it already created, so the Certificate also needed acme.cert-manager.io/http01-override-ingress-name set by hand. Once that was set the renewal passed in under a minute. If I hadn't forced the renewal, the first I'd have heard about it was an expiring certificate alert a month later.

Since then we've upgraded cert-manager, and with ingressClassName: haproxy set on its solver those manual overrides aren't needed anymore. Next up is rebuilding our bot handling on top of stick tables, which is the whole reason I wanted the real client IP in the first place.