Password Protecting a Redis Pod on Google Kubernetes

Let's reduce the blast radius of a network intrusion for our Redis caching.

Caching so the front end feels fast is crucial to a site users actually want to use. We use Google Kubernetes to run our sites, and after trying many WordPress-based solutions decided to go with Redis as our caching layer.

Out of the box if you spin up a pod running redis:8 you can access it from inside your K8s cluster, but then anyone can access it inside that cluster. They can read, write, or even flush the whole instance. No, the pod isn't accessible from the internet, but that relies on the network perimeter holding.

That's not an accident of your setup either. Ask a running pod and you'll find the official image ships with protected-mode no and bind * -::*, so it listens on every interface and answers anyone who asks. Redis' own guard rail is off before you touch anything.

A compromised sidecar, a misbehaving CI runner with cluster access, an app with an SSRF bug — any of those turn "internal only" into "reachable". If the cache is the one thing in your stack with no second lock, that's where an attacker goes first.

To fix this we need to enable the --requirepass flag on your Redis pods so that only the sites with the Redis password can access the pod they've been given access to.

Don't put the password in the manifest

The obvious approach is a Kubernetes Secret with the password in it, referenced by both the Redis pod and the clients. That works, and it's better than nothing, but Secrets are base64, not encryption — anyone with read access to the namespace has the value. More practically, a Secret you create by hand is a Secret somebody eventually commits, or dumps into a backup, or pastes into a terminal that's being recorded.

I'd rather the password only exist in a secrets manager and arrive in the cluster on its own. On GCP that's Secret Manager plus the External Secrets Operator, which watches a custom resource and syncs the value into a native Kubernetes Secret for you.

The authentication between the two is the part worth getting right. ESO can use a service account key file, and you should not — that's a long-lived credential sitting in the cluster, which is the problem you're trying to solve, one layer down. Use Workload Identity instead, so the operator's Kubernetes service account is bound to a GCP service account and gets short-lived tokens with no static key anywhere.

Scope that GCP service account narrowly. Mine can read secrets named redis-* and nothing else, via an IAM condition:

gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:redis-secrets@my-project.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor" \
  --condition='title=redis-secrets-only,expression=resource.name.startsWith("projects/my-project/secrets/redis-")'

Without the condition, the operator can read every secret in the project — including ones that have nothing to do with caching. The condition costs you one line and means a compromise of the operator is a much smaller event.

Generate the password as hex

openssl rand -hex 32 | tr -d '\n' | \
  gcloud secrets create redis-cache --data-file=- --project=my-project

Both -hex and -base64 take a count of bytes, not output characters. So openssl rand -hex 32 and openssl rand -base64 32 are the same 32 random bytes — 256 bits of entropy — just written down differently. Hex writes them as 64 characters of [0-9a-f]. Base64 writes them as 44 characters from a set that includes +, / and =.

Take the hex. The strength is identical; what changes is how many things downstream can mangle the value. That password gets handled by, at minimum:

  • Kubernetes expanding $(REDIS_PASSWORD) into an args list
  • redis-cli -a on a command line
  • YAML, in at least two manifests
  • whatever your application language does when it reads the environment variable

+, / and = are exactly the characters those layers have opinions about. When one of them mangles the value you don't get an error — you get a Redis server whose password is subtly different from the one your client is sending, and an authentication failure you'll debug from the wrong end. Hex sidesteps the whole category.

Two smaller details in that command.

tr -d '\n' strips the newline openssl adds to its output. --data-file=- stores exactly the bytes it receives, so without this your password ends in a newline, and whether that matters comes down to whether a given client trims whitespace before sending. Some do, some don't.

The pipe keeps the password out of your shell history and out of ps, which is why it never gets assigned to a variable. Don't echo it to confirm it worked — confirm the version exists instead:

gcloud secrets versions list redis-cache --project=my-project

The manifests

Three objects. A ClusterSecretStore describing where secrets come from, an ExternalSecret describing which one you want, and the Redis Deployment itself.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: my-project
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: redis-cache
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-manager
    kind: ClusterSecretStore
  target:
    name: redis-cache
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: redis-cache

No credentials appear in either one, which is the entire point — both are safe to track in git.

The Deployment reads the synced Secret into an environment variable and expands it into the Redis arguments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rediscache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rediscache
  template:
    metadata:
      labels:
        app: rediscache
    spec:
      containers:
      - name: redis
        image: redis:8.10.0
        args:
        - --requirepass
        - $(REDIS_PASSWORD)
        env:
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-cache
              key: password
        ports:
        - containerPort: 6379

Two things here are deliberate.

args with $(VAR), not a command shell. Kubernetes expands $(REDIS_PASSWORD) from the container's own environment before exec'ing the entrypoint, so the password never passes through a shell. Wrapping this in sh -c "redis-server --requirepass $REDIS_PASSWORD" would put it in the process list, visible to anything that can read /proc in that container.

Pin the image tag. A bare image: redis means redis:latest, and latest is resolved when the pod is scheduled — not when you wrote the manifest. I've seen a pod that had been running untouched for years silently jump several major versions during an unrelated node upgrade, because it rescheduled and re-pulled. If you're deliberately adding authentication to something, you don't want its major version changing underneath you on the next node roll.

Your client then reads the same Secret:

- name: CACHE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: redis-cache
      key: password

Verify it, don't assume it

Applying the manifest doesn't mean authentication is on, and both ways it can go wrong are silent. If the Secret key exists but holds an empty value, --requirepass "" starts a server with no password at all. If the environment variable doesn't exist, Kubernetes leaves the reference unchanged and your password becomes the literal string $(REDIS_PASSWORD). Neither one complains. Check the running server, both directions:

# From outside, with no credentials — this should be refused
kubectl exec deploy/rediscache -- redis-cli -h rediscache ping
# NOAUTH Authentication required.

# From inside, using the pod's own environment — this should work
kubectl exec deploy/rediscache -- \
  sh -c 'redis-cli --no-auth-warning -a "$REDIS_PASSWORD" ping'
# PONG

The first command is the one that matters. NOAUTH is the healthy answer, and it's the only real evidence you have that the thing is enforcing anything.

The ordering problem that will bite you

This is the part I'd have liked to know in advance, and it generalizes well beyond Redis.

A lot of caching clients do setup work at startup. They connect, check the server, install a drop-in, register themselves. If that startup step can't authenticate, a well-behaved client logs an error and carries on without the cache. It doesn't crash. Your app stays up, serves correct pages, and quietly does all the expensive work on every single request.

The WordPress object cache plugin we use does exactly this: it runs an enable step on every container start, and if it can't reach an authenticated Redis it declines to install the drop-in and the site runs uncached until something restarts it. Nothing is on fire. The site is just slow, and only for logged-in users, because anonymous traffic is served by a CDN that hides the symptom completely.

So the sequence matters:

  1. Stand up the Redis pod with the password required.
  2. Confirm it answers NOAUTH.
  3. Then point clients at it, with the host and the password set in the same change.

Setting the host in one deploy and the password in the next gives you a window where the client connects unauthenticated, fails, and disables its own cache. Do it in one patch.

Rotation is where the design shows

The password now exists in three places that update at different times, and understanding that is most of what you need to rotate it:

  • Secret Manager. Adding a version changes nothing by itself.
  • The Kubernetes Secret. ESO syncs it on its refreshInterval, which the config above sets to an hour.
  • The running pods. They read the Secret into environment variables once, at startup. A running pod keeps the password it booted with until it restarts.

That last one catches people. Updating the Secret does not touch a running pod. So a rotation is: add a version, force ESO to sync, restart Redis, then restart every client.

You can force the sync rather than waiting an hour by changing any annotation on the ExternalSecret:

kubectl annotate externalsecret redis-cache force-sync="$(date +%s)" --overwrite

Watch out for the window between the sync and the Redis restart. During it, the Secret holds the new password while Redis still requires the old one — so any client that happens to restart right then comes up with the wrong credential and, per the section above, may quietly disable its own cache. Keep that window short, and don't rotate in the middle of an unrelated deploy.

Also worth planning for: a Redis pod with no volume loses everything when it restarts, and a rotation restarts it. Every client sharing that pod goes cold at the same moment. That's fine if your cache is genuinely a cache, but "fine" assumes your app can survive a cold start under real traffic. Warm it deliberately afterwards rather than finding out.

Was it worth it

It's not a huge change, about 15 lines of YAML, but it reduces the blast radius of any intrusion, limiting the amount of cleanup we'd need to do if a malicious actor did get past our network perimeter.