September 7, 2026
Station 4: The Day Before the Six Minutes
Live Wire: A Solo GitLab Upgrade Log — Tomorrow, the database engine itself gets swapped out from underneath a live instance. Today is twelve verification items and a full rehearsal against a real-data copy.

The riskiest station in the whole plan, and a full day spent trying to make it boring
This is Part 8 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 3: GitLab Behaved, I Didn’t | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 5–6: The Migration That Fixed Itself
TL;DR
Station 4 swaps the database engine itself — PostgreSQL 15 to 16 — the riskiest kind of change in the whole plan, because it touches how data is stored, not just which application code reads it. The day before, a full risk review turned up three separate landmines that would otherwise have surfaced mid-downtime: an unreachable package source, a version-detection script that sorts strings instead of numbers, and an automatic rollback that covers far less than it sounds like it does. A full rehearsal against a real-data copy caught one more thing the plan had missed entirely. The actual cutover, the next day, took six minutes against an official estimate of thirty.
The Riskiest Kind of Change
Every station so far has upgraded GitLab’s application code. This one upgrades the thing underneath it — the database engine itself, PostgreSQL 15 to 16. A failed application upgrade is usually recoverable. A failed database engine conversion, mid-way, can leave the data directory in a state that’s hard to describe as anything other than broken. That difference is what justified spending an entire day on verification and rehearsal before touching anything for real.
The Fix, in One Sentence
Spend the risk on the day before the maintenance window, not during it — verify everything that could fail silently, and rehearse the actual conversion against a real copy of the data before doing it against the real thing.
Quick Start: The Two Checks Worth Running First
# Confirm the upgrade script's package source is actually reachable
# from this host — don't assume it, given more general connectivity works
curl -v --max-time 10 https://apt.postgresql.org/ 2>&1 | tail -20
curl -v --max-time 10 https://archive.ubuntu.com/ 2>&1 | tail -5
# Simulate the version-detection logic against the real data directory
# before trusting it during an actual maintenance window
find "${PG_HOME}"/+([0-9])?(.[0-9])/main -maxdepth 1 -name PG_VERSION \
| grep -v "$PG_VERSION" | sort -r | head -n1
Both of these caught something real.
Finding 1: The Upgrade Script’s Own Package Source Was Unreachable
The upgrade script’s core dependency:
if ! ( apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
postgresql-${PG_OLD_VERSION} postgresql-client-${PG_OLD_VERSION} ) >/dev/null; then
echo "ERROR! Failed to install PostgreSQL ${PG_OLD_VERSION}. Exiting..."
exit 1
fi
Testing connectivity ahead of time found that this host couldn’t reach apt.postgresql.org at all — every request to its Fastly-fronted IPs (151.101.x.x) timed out — while archive.ubuntu.com, github.com, and google.com all worked normally. Discovering this during the actual maintenance window would have meant the upgrade script failing at this exact line, with the service already stopped.
The fix was a purpose-built image with the dependency baked in ahead of time:
FROM kkimurak/sameersbn-postgresql:16-20260820
COPY postgresql-client-15_15.19-1.pgdg24.04+2_amd64.deb /tmp/
COPY postgresql-15_15.19-1.pgdg24.04+2_amd64.deb /tmp/
RUN dpkg -i /tmp/postgresql-client-15_*.deb /tmp/postgresql-15_*.deb && rm -f /tmp/*.deb
RUN sed -i '/apt.postgresql.org/d' /etc/apt/sources.list
The .deb packages were downloaded from a machine that could reach the official source, then transferred over — sidestepping the host's connectivity gap entirely rather than trying to fix it.
Finding 2: A Script That Sorts Strings, Not Numbers
The upgrade script finds the old PostgreSQL version by scanning the data directory:
PG_OLD_VERSION=$(find ${PG_HOME}/+([0-9])?(.[0-9])/main -maxdepth 1 -name PG_VERSION \
| grep -v $PG_VERSION | sort -r | head -n1 | cut -d'/' -f5)
The data directory here held both a three-year-old leftover 12/ folder and the current 15/ — and sort -r sorts strings, not numbers. Testing it directly against the real directory confirmed it worked correctly in this specific case:
---FIND_RESULT---
/var/lib/postgresql/12/main/PG_VERSION
/var/lib/postgresql/15/main/PG_VERSION
---AFTER_GREP_SORT---
DETECTED_OLD_VERSION=15 ← correct
It worked here because "15" sorts after "12" both as strings and as numbers — the two orderings happen to agree at this range. That's a coincidence of the current version numbers, not a guarantee: if a single-digit version like 9.x ever showed up in that same directory, string sort would place it after "15" and "12", picking the wrong one. Worth knowing the boundary of a check even when the check passes.
Finding 3: Automatic Rollback Covers Less Than It Sounds Like
Reading through the upgrade script’s failure handling surfaced a narrower safety net than expected: only the “install the old binary” step has automatic recovery. The steps that actually convert the data — initdb and pg_upgrade — have no automatic protection if they fail partway through. A failure there could leave the data directory in a half-converted state with nothing built in to walk it back.
That gap is exactly why the day’s plan included **an additional logical backup via pg_dump, on top of the standard whole-instance backup — a second, independent way to reconstruct the data if the conversion step itself went wrong in a way the built-in tooling couldn't recover from.

Every finding here would have surfaced eventually. The only choice was when.
The Decisive Step: Rehearsing Against Real Data, Not Just Checking Boxes
Verifying preconditions is one level of confidence. Actually running the conversion is another. The day’s plan included copying /data/postgresql/15 — 4.5GB — into a temporary, isolated location and running the entire real pg_upgrade process against that copy, with zero impact on the actual production data directory or the live service.
Every compatibility check passed:
Checking for reg* data types in user tables ok
Checking for incompatible "aclitem" data type in user tables ok
Checking for not-null constraint inconsistencies ok ← new in PG16
Checking for presence of required libraries ok ← pg_trgm/btree_gist compatible
And the whole rehearsal — copy, convert, verify — took 98 seconds, well under the vendor’s own “within 30 minutes” estimate for the real thing.
What the Rehearsal Caught That the Plan Missed
Checking for extension updates notice
Your installation contains extensions that should be updated
with the ALTER EXTENSION command. The file
update_extensions.sql
The original plan only included vacuumdb as a post-upgrade step. It didn't include running the extension-update script at all — which would have left pg_trgm and btree_gist silently stuck on their PostgreSQL-15-era versions indefinitely. This is the entire argument for rehearsing instead of just reviewing documentation: a written plan doesn't tell you what it forgot. Running the actual process does.
The Next Day: 98 Seconds Became 6 Minutes
Downtime: 12:04:00 (stop) → 12:09:56 (GitLab back up)
btree_gist: 1.5 → 1.7
pg_trgm: 1.3 → 1.6
Six minutes of real downtime, against an official estimate of thirty. The extension updates the rehearsal had caught — absent from the original plan — ran successfully as part of the real cutover.
One more detail worth keeping in mind for any migration involving pg_dump: a file appearing on disk doesn't mean the job is done. Mid-process, a background task reported the dump file as already existing at 319MB, while the process was still actively writing to it — the final size came out to 375MB once it actually finished. File existence and process completion are two different signals, and only one of them is reliable.
Under the Hood
+-------------------------+---------------------------------------------+----------------------------------------------------+
| Item | What I Did | Why It Mattered |
+-------------------------+---------------------------------------------+----------------------------------------------------+
| Package source | Tested connectivity to apt.postgresql.org | Would have failed the upgrade script mid-downtime, |
| | ahead of time, built a custom image instead | with the service already stopped |
+-------------------------+---------------------------------------------+----------------------------------------------------+
| Version detection | Simulated the string-sort logic against the | Passed this time, but the boundary (single-digit |
| | real data directory before trusting it | versions) is now a known, documented risk |
+-------------------------+---------------------------------------------+----------------------------------------------------+
| Rollback coverage | Read the script's failure handling in detail | Found the real conversion steps have no automatic |
| | rather than assuming full rollback protection | recovery — justified an extra logical backup |
+-------------------------+---------------------------------------------+----------------------------------------------------+
| Full rehearsal | Ran the real pg_upgrade against an isolated | Caught a missing extension-update step the written |
| | real-data copy, not just a documentation review | plan never included |
+-------------------------+---------------------------------------------+----------------------------------------------------+
What Actually Worked
Every one of the three findings from the risk review would have surfaced eventually — the only real choice was whether that happened on a quiet afternoon with time to fix it, or during a live maintenance window with the service already down. That’s the entire case for spending a full day on this station’s preparation instead of treating it the same as the application-level upgrades before it. The rehearsal mattered even more than the review: a written plan can be internally consistent and still be missing a step nobody thought to write down, and the only way to find that kind of gap is to actually run the process once before running it for real.
Where This Still Falls Short
The rehearsal ran against an isolated copy of real data, not the live system — worth stating plainly so it’s clear this wasn’t testing directly against production. And the version-detection logic’s known weak spot (single-digit versions breaking string sort) wasn’t fixed here, just documented — it happened to be safe this time, and that boundary will need checking again the next time this same script runs against a data directory that might have grown a third version alongside the current two.
Have you ever found the real value of a rehearsal wasn’t confirming the plan was right, but discovering the step the plan never mentioned?
This is Part 8 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 3: GitLab Behaved, I Didn’t | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 5–6: The Migration That Fixed Itself
GitHub: jasonChen0604
Portfolio: jason-chen-1cb56.web.app
LinkedIn: jason-cj-chen