17 PARTS
Live Wire: A Solo GitLab Upgrade Log
Eight required stops, one CVSS 9.4 vulnerability, and no test environment to rehearse any of it in — upgrading a live, self-hosted GitLab alone.
01
The Plan: Upgrading a Live GitLab From 17.4.2 to 19.2.4 With Zero Test Environments
A GraphQL code-injection vulnerability, CVSS 9.4, forced a production GitLab instance three years and eight major versions behind current to finally upgrade —...
Aug 27
02
The Runner Audit: 43 Runners, One Column Everyone Trusts, One Wrong Answer
Before the upgrade path even started, the plan called for trimming the runner fleet — fewer moving parts to carry across eight version stops. The obvious...
Sep 1
03
Five Runners, Five Traps: Migrating Off a Deprecated Token, One Failure Mode at a Time
Legacy registration tokens are on their way out, and five active runners needed to move to the newer authentication-token model before the upgrade path could...
Sep 3
04
Ready the Foundation: 170GB Reclaimed and One Wrong Assumption Corrected Before the Upgrade Started
Before any of the eight required version stops, there was a day of unglamorous groundwork: confirming the backup mechanism actually produces a restorable file,...
Sep 3
05
The Image That Wouldn't Die: Tracing a Stuck Deletion to a Protocol-Level Lie
Deleting a container image repository through GitLab’s UI produced the expected “scheduled for deletion” message — and then nothing happened for days. The...
Sep 4
06
Station 1–2: The Estimation Trap
Station 1 finished its background migrations in minutes, which made Station 2 — carrying the same “ordinary risk” label — feel like it should behave the same...
Sep 4
07
Station 3: GitLab Behaved, I Didn't
Station 3–17.8.7 to 17.11.7 — was the cleanest version bump so far: database size actually shrank slightly from orphan-data cleanup, and the largest migrated...
Sep 7
08
Station 4: The Day Before the Six Minutes
Station 4 swaps the database engine itself — PostgreSQL 15 to 16 — the riskiest kind of change in the whole plan, because it touches how data is...
Sep 7
09
Station 5–6: The Migration That Fixed Itself
Official advice, before Station 5 even started: don't fix this one, just keep upgrading. It worked. Verifying exactly how it worked is a different story.
Sep 9
10
Station 6: The Bug GitLab Hasn't Fixed Yet
The error message said a column was missing. It wasn't missing — it had been renamed three versions earlier. The real cause was a GitLab bug that's still open today.
Sep 9
11
Tuning the Parallelism Knob
Stations 5 through 8 — A database setting that isn't even in the official API sped up mid-size migrations noticeably. A 320,000-row table didn't care.
Sep 10
12
Station 7–8: The Backup That Looked Broken
A backup file appeared on disk at a third of its usual size. The obvious read was failure. The actual answer was: it just wasn't done writing yet.
Sep 11
13
Reaching 19.2.4: The Milestone
Ten mandatory stations after a CVSS 9.4 vulnerability announcement, the version number that started this whole series is finally the version actually running.
Sep 12
14
Two Thousand Stalled Jobs
Ten stations into this upgrade, an unrelated number showed up in the admin panel: over 2,800 dead jobs, the oldest one six months old.
Sep 13
15
Station 11, Part 1: Three Blockers a Dry-Run Almost Missed
The plan said “just swap the image, keep the environment variables.” A dry-run found three separate reasons that assumption was wrong, stacked one on top of the next.
Sep 14
16
Station 11, Part 2: 202, Not 400 — The Image That Wouldn't Die, Finally Does
58,818 failed requests, back in August. One API call, one month later, that finally returns what it was always supposed to.
Sep 15
17
Discipline, Not Luck
Sixteen posts, eleven stations, two CVEs, one long-standing bug — all closed. Looking back, one habit shows up in almost every single entry.
Sep 16