JC_

September 1, 2026

Live Wire: A Solo GitLab Upgrade Log — Series Overview

Gitlab
DevOps
Self Hosted
Site Reliability
Software Engineering

Eight required stops, one CVSS 9.4 vulnerability, and no test environment to rehearse any of it in.

This GitLab instance has been carrying production traffic for three years — self-hosted, docker-compose deployment, Postgres and a container registry underneath it, real merge requests and real CI pipelines from people who don’t stop for maintenance windows. There’s no staging replica of it anywhere. There never has been.

This series is the record of moving that instance from GitLab 17.4.2 to 19.2.4 — eight mandatory version stops, one CVSS 9.4 vulnerability that forced the timeline, and no test environment to rehearse any of it in advance. Every entry follows roughly the same shape: what the actual risk was, what got checked before anything ran, and what happened when the plan met the real system.

Below is every entry, in the order things actually happened. The series is complete — seventeen parts from the first CVSS 9.4 alert to the closing retrospective.

Part 1 — The Plan: Upgrading a Live GitLab From 17.4.2 to 19.2.4 With Zero Test Environments

A GraphQL vulnerability with a CVSS score of 9.4 doesn’t wait for a convenient window. Mapping eight required stops with no test environment, and the one silent bug in the plan that could have quietly destroyed data instead of just breaking a build.

Part 2 — The Runner Audit: 43 Runners, One Column Everyone Trusts, One Wrong Answer

Forty-three CI runners, and a completely reasonable way to tell which ones were still in use — that turned out to be measuring the wrong thing entirely.

Part 3 — Five Runners, Five Traps: Migrating Off a Deprecated Token, One Failure Mode at a Time

Migrating five runners off a deprecated authentication token. Every single one of them failed for a different reason.

Part 4 — Ready the Foundation: 170GB Reclaimed and One Wrong Assumption Corrected Before the Upgrade Started

The unglamorous work before any real risk starts: backup verification, disk cleanup, and a log file that was quietly growing by 9GB a day.

Part 5 — The Image That Wouldn’t Die: Tracing a Stuck Deletion to a Protocol-Level Lie

GitLab said the image was deleted. Three days later, it still wasn’t. The actual cause lived one layer below the application, in a handshake nobody thinks to check.

Part 6 — Station 1–2: The Estimation Trap

Two upgrade stations, both labeled “ordinary risk.” One finished in six minutes. The other one, by the first estimate, was going to take twenty hours.

Part 7 — Station 3: GitLab Behaved, I Didn’t

Zero surprises from GitLab itself. Four separate ones from sudo, shell escaping, and a grep filter that quietly ate its own output.

Part 8 — Station 4: The Day Before the Six Minutes

The single riskiest step in the whole plan: swapping the database engine underneath a live instance. One full day of rehearsal turned a 30-minute official estimate into six minutes of real downtime.

Part 9 — Station 5–6: The Migration That Fixed Itself

A migration flagged to fail before it even ran. The official advice: don’t fix it, just keep upgrading — and verifying exactly how that turned out to be true.

Part 10 — Station 6: The Bug GitLab Hasn’t Fixed Yet

An intermittent 500 pointing at a column that vanished. Except it hadn’t vanished — it had just been renamed three versions earlier, and the real cause traced back to a GitLab bug that’s still open today.

Part 11 — Tuning the Parallelism Knob

A database setting that isn’t even in the official API sped up mid-size migrations noticeably. A 320,000-row table didn’t care.

Part 12 — Station 7–8: The Backup That Looked Broken

A backup file appeared on disk at a third of its usual size. The obvious read was failure. The actual answer was: it just wasn’t done writing yet.

Part 13 — Reaching 19.2.4: The Milestone

Ten mandatory stations after a CVSS 9.4 vulnerability announcement, the version number that started this whole series is finally the version actually running.

Part 14 — Two Thousand Stalled Jobs

Ten stations into this upgrade, an unrelated number showed up in the admin panel: over 2,800 dead jobs, the oldest one six months old.

Part 15 — Station 11, Part 1: Three Blockers a Dry-Run Almost Missed

The plan said “just swap the image, keep the environment variables.” A dry-run found three separate reasons that assumption was wrong, stacked one on top of the next.

Part 16 — Station 11, Part 2: 202, Not 400 — The Image That Wouldn’t Die, Finally Does

58,818 failed requests, back in August. One API call, one month later, that finally returns what it was always supposed to.

Part 17 — Discipline, Not Luck

Sixteen posts, eleven stations, two CVEs, one long-standing bug — all closed. Looking back, one habit shows up in almost every single entry. This is the final entry in the series.