Catching Missed WordPress Database Migrations with a Kubernetes Watcher

Let's implement automated database upgrade scanning across 100+ WordPress sites that catches plugin database migrations and lets me know when things get missed.

Sometimes when you update a WordPress plugin, database migrations need to run. If you're managing a few sites, it's easy enough to sign in and run them manually and verify that they've run simply by looking at the site. I run 100+ sites though spread across a kubernetes cluster which means I need to have a system setup to catch database migrations and alert me that they need to be run. It also needs to note when a migration got stuck and needs to be manually addressed.

All this worked was prompted by me finally noticing that 41 of our 100+ sites silently missed a migration that would give us a boost in speed for WP Stateless. I ran this migration manually when I noticed it 2 years ago, but clearly hadn't noticed that it silently failed on some of our infrastructure until I happened to look at just the right site to see the failure.

Yup it would have been easy to just run the migration again on the single site where I found the failure. That would still mean I didn't know which other sites didn't run the migration? What if some of those sites failed again and need a deeper look? What about migrations from other plugins that didn't run that I haven't caught yet?

Today we'll walk through our system that watches for database migrations, and if it finds them it creates a Github issue so I can investigate them and run any that need to be run.

Two different failures

We had two different failure types that needed to be addressed individually.

Detection — we didn't know the plugin shipped migrations at all. Nothing in our update process asked the question.

Reconciliation — the migrations did run, on most sites. They failed on a subset. Nothing checked whether the outcome matched the intent.

The obvious fix for the first one is a checklist item in our plugin-update issue template: "check whether this release adds a migration." That's worth doing, and I did it. But it does nothing for the second failure. A checklist only fires when someone opens an issue, and it asks a human to remember to verify 150 sites by hand. Nothing was actually broken in our process at update time. Our update scripts ran during our release, they just silently failed on some sites sometimes.

That's a hard issue for a human reviewer to catch.

The registry approach, and why I abandoned it

My first design was a curated list of plugins known to ship migrations, checked on every version bump. Straightforward, easy to explain, but it has a few problems that render it unworkable.

It goes stale. A plugin with no migrations today can add one in any future release. The registry silently stops covering it, and you don't find out until the same thing happens again.

Building it is unreliable. I scanned all 78 plugins in our install for the usual signals — a migrations/ directory, a DB_VERSION constant, use of WP_Background_Process. Eleven hits. Seven of them were vendor/phpunit/.../Migration directories that have nothing to do with WordPress. The real answer was four plugins.

Four. Out of a fleet where I'd later find roughly fifteen plugins tracking migration or schema state.

Detect at the state layer instead

Instead of trying to track different migration infrastructure in plugins a better way to track was to query wp_options for some keywords that plugins use to track their database and migration versions.

Every plugin that versions its schema writes that version somewhere. In WordPress that's almost always an option row. So sweep them:

SELECT option_name, option_value FROM wp_options
WHERE option_name REGEXP '(db_version|schema_version|_migration|migrations)'
  AND option_name NOT LIKE '_transient%'
  AND option_name NOT LIKE '_site_transient%'

Run that on every site, group by option name, and take the most common value as the fleet consensus. Anything that disagrees is drift.

This needs no registry and no knowledge of any plugin's internals. On our fleet it surfaced 36 distinct options across about fifteen plugins — Gravity Forms, Action Scheduler, Yoast, Site Kit, Pods, Auth0, WP Mail SMTP, and others I'd never have thought to put on a list.

Read the database, not the option API

One implementation detail that turned out to matter more than I expected. The probe reads wp_options through $wpdb rather than calling get_option().

Two reasons. get_option() runs through filters, so a plugin can rewrite what you see. More importantly it goes through the object cache. Since we run Redis it's possible that a scan could get a stale/cached value if we use get_option().

Normalization is most of the work

The raw values aren't comparable, and getting this wrong produces a tool nobody trusts.

Timestamps. One plugin stores its migration state as a serialized array containing started and finished unix timestamps. Every site has different values by construction. Hashing the raw value reported 150 of 152 sites as outliers. The fix is to walk the unserialized structure and drop timing fields before comparing — we care whether a migration finished, not when.

Equivalent terminal states. Twenty sites recorded a migration as skipped rather than finished. That's what the plugin writes when its should_run() check returns false: the migration didn't apply to that site. It's every bit as done as finished, but against a finished consensus it reads as drift. I collapse finished|complete|completed|done|skipped|not_required to a single token.

Absence is not the same as a mismatch. A site with a different version is unambiguous: same plugin, different state. A site missing the option entirely usually just means the plugin isn't active there. Without a plugin-to-option map — the registry I'd just rejected — you can't tell those apart automatically. So I report them as separate sections and only treat value mismatches as actionable.

What it actually found now that the noise is dealt with

Once I was pretty sure I had valid issues and not just noise, I could run the check our sites. Here's what each turned out to be:

Gravity Forms, 29 production sites. Real, and stuck permanently. The plugin calls its upgrade routine on every non-AJAX request, so these sites had been retrying constantly for months. Every attempt died in the same place:

$lock_params = $this->get_upgrade_lock();
if ( $lock_params && ! $force_upgrade ) {
    // Abort. Upgrade already in process.
    return false;
}

That lock is only cleared by a routine that's itself conditional:

if ( $to_version != $versions['version'] ) {
    $this->clear_previous_upgrade();
}

All 29 sites carried a lock recording to_version as the version that was already installed. So the clear condition was never true, the lock never lifted, and the upgrade could never start. A self-perpetuating deadlock left behind by one interrupted first attempt.

Google Site Kit, three sites. One real — a site that hadn't had an admin page load since the plugin updated, because Site Kit hooks its migrations to admin_init. Notably that one would have self-healed on its own the next time someone opened wp-admin. The other two just had the plugin deactivated, so admin_init will never fire and the version will sit unchanged forever. Not drift.

WP Migrate DB. Not drift. The plugin had been uninstalled fleet-wide years ago and left its options behind. The sites "behind" the consensus were cleaned up at different times; the "correct" value was no more correct than the rest.

An old WP-Stateless option. Not drift, same shape. Nothing in the entire install references it any more. I proved it by comparing the table it appeared to version across sites reporting three different values — byte-identical schemas. The number tracks nothing.

Yoast, one site. Not a migration problem at all. That site was running Yoast Premium as its active SEO plugin with the free version inactive — the inverse of every other site — so its migration state froze wherever Premium left it.

One in five

That ratio is the most useful thing this exercise produced. Of five anomalies, one was a migration that needed running. The rest were two orphaned option families, an inactive plugin, and a plugin version mismatch.

Most drift is archaeology, not breakage.

It changed how I built the reporting. The automated issue leads with a triage checklist — is the plugin even installed, is it active, what does the upgrade actually do, what triggers it, is anything blocking it — rather than implying the fix is to run a migration. A tool that cries wolf four times out of five gets ignored by the sixth month, and then you're back where you started with a nag notice nobody reads.

Getting the check inside 150 sites

Every site is a pod, and the database is only reachable from inside it. So the audit finds the running pod for a site and pipes a PHP script into its own WP-CLI:

pod=$(kubectl -n "$ns" get pods --selector=app="$app" \
      --field-selector=status.phase=Running \
      -o jsonpath='{.items[0].metadata.name}')

kubectl -n "$ns" exec -i "$pod" -- \
  wp --allow-root eval-file - --skip-plugins --skip-themes < probe.php

eval-file - reading from stdin is doing real work there. My first attempt used wp db query with the SQL inline, and I spent an embarrassing amount of time on quoting. A query containing REPLACE(x, CHAR(10), '') came back as Success: Query succeeded. Rows affected: -1 with no rows — the empty-string literals were being eaten somewhere between my shell, kubectl exec's argv handling, and WP-CLI. Piping a script over stdin sidesteps every layer of that, and it means the normalization can be written in PHP where it belongs instead of as increasingly baroque SQL.

--skip-plugins --skip-themes keeps the probe from booting 50 plugins on every one of 150 sites just to read a few option rows.

Why it runs in the cluster

I wanted this on a schedule without my interaction so it runs as a Kubernetes CronJob with its own ServiceAccount, which needs no interactive credential at all.

The ServiceAccount

The watcher needs to exec into pods across two namespaces.

What it actually needs is three verbs:

rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["list"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]

Bound with a Role in each namespace rather than a ClusterRole. The watcher has no reason to reach kube-system or anything else, and a ClusterRole would have been one line shorter and considerably worse.

pods/exec is effectively root-in-container for that site. It's narrower than what I already hold as an operator, but mine is interactive and this one sits there permanently, so it's worth deciding deliberately if you're comfortable with this rather can just accepting what I've done.

kubectl auth can-i will lie to you

Having written the RBAC carefully, I went to verify it:

kubectl auth can-i create pods/exec -n prod --as=system:serviceaccount:default:migration-watcher
# yes

Great. Then the negative cases, which should all have been no:

kubectl auth can-i get secrets -n prod --as=...        # yes
kubectl auth can-i get pods -n kube-system --as=...    # yes

That's alarming until you try it with a service account that doesn't exist:

kubectl auth can-i delete nodes --as=system:serviceaccount:default:definitely-not-real
# yes

The impersonation wasn't being evaluated at all, so every answer was really about my permissions, not the service account's. I don't know whether that's a quirk of our cluster's setup or a more general trap, but either way auth can-i --as= gave me a confident wrong answer in the direction that matters — it told me a permission existed when I was trying to prove it didn't.

The check that actually works is enumerating what references the account:

kubectl get clusterrolebindings -o json | jq -r --arg n migration-watcher \
  '.items[] | select(.subjects[]?|.name==$n) | .metadata.name'

kubectl get rolebindings -A -o json | jq -r --arg n migration-watcher \
  '.items[] | select(.subjects[]?|.name==$n) | "\(.metadata.namespace)/\(.metadata.name)"'

Two RoleBindings, no ClusterRoleBinding. That I believe, because it's reading the objects rather than asking the API to reason about them.

Automating it without creating noise

The check runs weekly as a Kubernetes CronJob and maintains exactly one rolling issue.

The rules I settled on:

  • Drift found, no open issue → create one
  • Drift found, same set as last run → do nothing
  • Drift set changed → rewrite the issue body, add one comment saying what resolved and what appeared
  • No drift, issue open → close it

That second rule is the one that makes it liveable. A quiet weekend produces no notifications at all. The script keeps no state of its own — the fingerprint of the current drift set lives in an HTML comment in the issue body, so the issue is the state.

It also rewrites only the region between two marker comments, and preserves anything written after them. If a tool is going to edit an issue on a recurring schedule, it needs to be safe to add your own notes to.

Watch your own watcher

The last piece: the whole thing authenticates with a GitHub token, and GitHub caps fine-grained tokens at 366 days. When that token expires the CronJob starts failing, and a failing CronJob is silent. One day I'd wake up and realize I hadn't seen any notifications of failed migrations, not because they hadn't happened but because my token wasn't valid and issues weren't being created.

So the watcher watches its own token. GitHub returns the expiry in a response header on any authenticated request, so it costs nothing extra:

Github-Authentication-Token-Expiration: 2027-08-25 15:20:13 UTC

Inside 45 days it files a separate rolling issue with the renewal steps in the body, keeps it updated as the date approaches, and closes it once it sees a renewed token.

You can shoot yourself in the foot here though. If you ignore the token expiration notice for 45 days it will expire and then not be able to update any issues, including the expiration issue. The issue is posted to our regular task board though so I should catch this in the regular course of checking tasks.

What I'd take to another stack

Very little of this is WordPress-specific.

  • Detect at the state layer, not the source layer. Asking "what do all the machines believe" scales and stays current. Asking "what should be true according to a list I maintain" decays from the day you write it.
  • Consensus needs a control. Something you already know is uniform, so you can tell a working check from one producing noise.
  • Normalization is the work. Timestamps, equivalent-but-differently-spelled states, and present-vs-absent all need handling before the output means anything.
  • Anomaly detection and diagnosis are different jobs. Mine is good at the first and useless at the second, and the reporting says so out loud.
  • Don't suppress to reduce noise. Reorder, group, annotate — but the moment you hide, you'll hide something real.
  • Whatever monitors the system needs its own liveness signal. A silent failure in a monitoring tool is worse than no tool, because now you think you're covered.
  • Verify permissions by reading the objects, not by asking. auth can-i --as= told me a permission existed when I was trying to prove it didn't. Any check that can confidently answer for an account that doesn't exist isn't a check.

The fleet is clean now, apart from a handful of staging sites I've explicitly excluded with the reason recorded next to each one. That last part matters more than it sounds — an exclusion without a reason is just a hidden bug, and in six months nobody will remember which it was.