September 9, 2026
Station 6: The Bug GitLab Hasn't Fixed Yet
Live Wire: A Solo GitLab Upgrade Log — The error message said a column was missing. It wasn’t missing — it had been renamed three versions earlier. The real cause was a GitLab bug that’s still open today.

A five-step trace from a generic 500 page to an open GitLab issue number
This is Part 10 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 5–6: The Migration That Fixed Itself | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Tuning the Parallelism Knob
TL;DR
After upgrading to 18.5.7, pages started throwing intermittent, generic 500 errors — easy to write off as noise, since not every request hit it. The real error underneath pointed at a missing database column. The column wasn’t missing; it had been renamed years earlier, and the table it belonged to was structurally fine. The actual cause traced back to a GitLab bug, still open today, where an unrelated migration triggers a stale in-memory schema cache. The almost-wrong fix was patching the database by hand. The actual fix was a restart.
A 500 Page That Comes and Goes
Not every failure announces itself clearly. After upgrading to 18.5.7 — with the container reporting healthy — the liveness endpoint and a handful of other pages started intermittently returning a plain, unhelpful “500: We’re sorry, something went wrong on our end.” Intermittent failures are the easiest kind to dismiss, precisely because most requests succeed and the pattern doesn’t jump out immediately.
The Fix, in One Sentence
Trace the error past its surface message to what the schema actually looks like, confirm whether it’s a real corruption or a stale cache, and resist the urge to “fix” a database that was never actually broken.
Quick Start: Quantifying and Confirming Before Acting
# Don't judge severity from a single screen refresh — count it
grep -c "PG::UndefinedColumn" gitlab-rails/production.log
-- Confirm whether the column is actually gone, or just renamed
SELECT column_name FROM information_schema.columns
WHERE table_name = 'application_settings'
AND column_name LIKE '%secret%';
Both of these changed the read on what was actually happening.
Step 1 and 2: A Vague Symptom, Then the Real Error
The screen said nothing useful. The logs said something specific:
ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR: column application_settings.pre_receive_secret_detection_enabled does not exist
LINE 1: ..."enable_artifact_external_redirect_warning_page", "applicati...
A missing column, on the face of it — the kind of error that usually means a migration didn’t run, or the schema is out of sync with the code expecting it.
Step 3: This Wasn’t a One-Off
grep -c "PG::UndefinedColumn" gitlab-rails/production.log
# 1506
1,506 occurrences, not a single unlucky request. Whatever this was, it was happening constantly, just not on every single page load — which is exactly the pattern that makes an intermittent error easy to underestimate from the UI alone.
Step 4: The Schema Wasn’t Actually Broken
SELECT column_name FROM information_schema.columns
WHERE table_name = 'application_settings' AND column_name LIKE '%secret%';
pre_receive_secret_detection_enabled genuinely wasn't there. But a different, clearly related column was: secret_push_protection_available — present, intact, doing its job. That's the pattern of a column that was renamed at some point, not one that vanished from a broken migration. The table itself was structurally fine.

The schema wasn’t broken. Something else was reading it wrong.
Step 5: An Open Issue With GitLab’s Name on It
Tracing the exact error signature further led to GitLab issue #576490 — open, no milestone attached, meaning not yet fixed on GitLab’s side at the time of writing. The description matched exactly: during the 18.2.8-to-18.4.2 range, a post_migrate migration called FinalizeMarkAdminBotRunnersAsHosted — a migration entirely about runners, with nothing to do with application_settings — runs, and as a side effect, Rails loads the ApplicationSetting model with a stale in-memory schema cache still holding the old, pre-rename column name.
The Fix That Almost Happened
The instinct, once a “missing column” error is confirmed, is to add the column back. That instinct was wrong here, and it’s worth being specific about why:
The migration actually throwing the error had nothing to do with application_settings — it was collateral damage from an unrelated runner migration
The replacement column, secret_push_protection_available, genuinely existed — proof the schema itself wasn't corrupted
Manually adding the old column back would have created an unofficial ghost column, one that could directly conflict with a real GitLab migration touching the same table later
Patching the database here would have “fixed” a database that was never actually broken, while creating a new, self-inflicted problem in its place.
The Actual Fix: Don’t Touch Data, Clear the Cache
ssh Gitlab "bash -ic 'cd ~/gitlab && echo \$SUDO_PASS | sudo -S docker-compose restart gitlab'"
A restart clears the stale in-memory schema cache without touching a single row of actual data. After restarting: the error count held steady at 1,506 — no new occurrences — and manually testing several pages (sign_in, explore, admin_settings) came back clean, with the liveness endpoint returning a consistent 200.
Under the Hood
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Step | What I Found | Why It Mattered |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Symptom | Generic, intermittent 500 — easy to dismiss | Underestimating an intermittent error is the |
| | as noise | most common way to let a real one slip past |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Real error | PG::UndefinedColumn on application_settings | Pointed at a specific, checkable claim instead of |
| | | a vague failure |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Scale | 1,506 occurrences via log grep | Confirmed this was systemic, not a one-off |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Schema check | Old column gone, but a renamed replacement | Ruled out real schema corruption before |
| | column genuinely existed | considering any fix |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Root cause | GitLab issue #576490 — an unrelated migration | Confirmed this wasn't a local misconfiguration, |
| | triggering a stale schema cache, still open | it was an upstream, already-known bug |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Fix | Restarted the service instead of editing the | Cleared the stale cache without creating an |
| | database | unofficial, conflict-prone ghost column |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
What Actually Worked
Quantifying the error before reacting to it — 1,506, not “sometimes” — is what turned a dismissible-looking symptom into something worth fully tracing. Checking the schema directly, rather than trusting the error message’s framing of “missing,” is what caught that the real problem was a stale cache, not a broken table — and that distinction is exactly what separated the correct fix (restart) from the tempting wrong one (patch the database).
Where This Still Falls Short
This is GitLab’s own bug, not a misconfiguration in this environment — worth stating plainly so it doesn’t read as an operational mistake. At the time of writing, issue #576490 is still open with no milestone attached, meaning GitLab hasn’t shipped an official fix. If that changes later, this account of “GitLab hasn’t fixed it yet” will need revisiting — the workaround here is a safe way to live with the bug, not a substitute for the real fix landing upstream.
Have you ever traced an error message back to a root cause that was completely unrelated to what the message itself claimed was wrong?
This is Part 10 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 5–6: The Migration That Fixed Itself | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Tuning the Parallelism Knob
GitHub: jasonChen0604
Portfolio: jason-chen-1cb56.web.app
LinkedIn: jason-cj-chen