JC_

← All series

17 PARTS

Live Wire: A Solo GitLab Upgrade Log

Eight required stops, one CVSS 9.4 vulnerability, and no test environment to rehearse any of it in — upgrading a live, self-hosted GitLab alone.

01

The Plan: Upgrading a Live GitLab From 17.4.2 to 19.2.4 With Zero Test Environments

A GraphQL code-injection vulnerability, CVSS 9.4, forced a production GitLab instance three years and eight major versions behind current to finally upgrade —...

Aug 27

02

The Runner Audit: 43 Runners, One Column Everyone Trusts, One Wrong Answer

Before the upgrade path even started, the plan called for trimming the runner fleet — fewer moving parts to carry across eight version stops. The obvious...

Sep 1

03

Five Runners, Five Traps: Migrating Off a Deprecated Token, One Failure Mode at a Time

Legacy registration tokens are on their way out, and five active runners needed to move to the newer authentication-token model before the upgrade path could...

Sep 3

04

Ready the Foundation: 170GB Reclaimed and One Wrong Assumption Corrected Before the Upgrade Started

Before any of the eight required version stops, there was a day of unglamorous groundwork: confirming the backup mechanism actually produces a restorable file,...

Sep 3

05

The Image That Wouldn't Die: Tracing a Stuck Deletion to a Protocol-Level Lie

Deleting a container image repository through GitLab’s UI produced the expected “scheduled for deletion” message — and then nothing happened for days. The...

Sep 4

06

Station 1–2: The Estimation Trap

Station 1 finished its background migrations in minutes, which made Station 2 — carrying the same “ordinary risk” label — feel like it should behave the same...

Sep 4

07

Station 3: GitLab Behaved, I Didn't

Station 3–17.8.7 to 17.11.7 — was the cleanest version bump so far: database size actually shrank slightly from orphan-data cleanup, and the largest migrated...

Sep 7

08

Station 4: The Day Before the Six Minutes

Station 4 swaps the database engine itself — PostgreSQL 15 to 16 — the riskiest kind of change in the whole plan, because it touches how data is...

Sep 7

09

Station 5–6: The Migration That Fixed Itself

Official advice, before Station 5 even started: don't fix this one, just keep upgrading. It worked. Verifying exactly how it worked is a different story.

Sep 9

10

Station 6: The Bug GitLab Hasn't Fixed Yet

The error message said a column was missing. It wasn't missing — it had been renamed three versions earlier. The real cause was a GitLab bug that's still open today.

Sep 9

11

Tuning the Parallelism Knob

Stations 5 through 8 — A database setting that isn't even in the official API sped up mid-size migrations noticeably. A 320,000-row table didn't care.

Sep 10

12

Station 7–8: The Backup That Looked Broken

A backup file appeared on disk at a third of its usual size. The obvious read was failure. The actual answer was: it just wasn't done writing yet.

Sep 11

13

Reaching 19.2.4: The Milestone

Ten mandatory stations after a CVSS 9.4 vulnerability announcement, the version number that started this whole series is finally the version actually running.

Sep 12

14

Two Thousand Stalled Jobs

Ten stations into this upgrade, an unrelated number showed up in the admin panel: over 2,800 dead jobs, the oldest one six months old.

Sep 13

15

Station 11, Part 1: Three Blockers a Dry-Run Almost Missed

The plan said “just swap the image, keep the environment variables.” A dry-run found three separate reasons that assumption was wrong, stacked one on top of the next.

Sep 14

16

Station 11, Part 2: 202, Not 400 — The Image That Wouldn't Die, Finally Does

58,818 failed requests, back in August. One API call, one month later, that finally returns what it was always supposed to.

Sep 15

17

Discipline, Not Luck

Sixteen posts, eleven stations, two CVEs, one long-standing bug — all closed. Looking back, one habit shows up in almost every single entry.

Sep 16