September 9, 2026
Station 5–6: The Migration That Fixed Itself
Live Wire: A Solo GitLab Upgrade Log — Official advice, before Station 5 even started: don’t fix this one, just keep upgrading. It worked. Verifying exactly how it worked is a different story.

Half the story checked out under verification. The other half didn’t — and that’s still worth telling
This is Part 9 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 4: The Day Before the Six Minutes | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 6: The Bug GitLab Hasn't Fixed Yet
TL;DR
Before Station 5 even started, official documentation already predicted a specific migration would fail — and gave counterintuitive advice: don’t fix it, just keep upgrading, because later versions clean it up automatically. It failed exactly as predicted. Continuing forward instead of stopping to patch it was the right call — the migration is now confirmed finalized and the data fully intact. Verifying that story for this article, though, turned up something the original research got wrong: the two specific "fix migrations" the advice pointed to don't exist anywhere in this environment. The outcome was real. The mechanism I wrote down for it wasn't provable.
A Migration That Was Supposed to Fail
Some failures are surprises. This one wasn’t. Before Station 5 (17.11.7 to 18.2.8) even started, official GitLab documentation and forum reports already flagged that BackfillSentNotificationsAfterPartition — a migration touching the sent_notifications table — was expected to fail during this exact version jump.
The Fix, in One Sentence
Trust documented, counterintuitive advice over instinct — and then verify the outcome for real before writing about it, rather than assuming the advice’s explanation was correct just because the advice itself worked.
Quick Start: Checking a Migration’s Real Status
-- Confirm whether a specific migration actually finished, and when
SELECT id, job_class_name, status, finished_at
FROM batched_background_migrations
WHERE job_class_name = 'BackfillSentNotificationsAfterPartition';
-- Confirm whether a named "fix" migration actually exists in this environment
SELECT id, job_class_name, status, finished_at
FROM batched_background_migrations
WHERE job_class_name IN ('MigrationNameOne', 'MigrationNameTwo');
The first query told the expected story. The second one didn’t.
What the Documentation Said, and Why It Made Sense
The official explanation, once tracked down: when 18.2.x schedules this backfill, it doesn’t first create partitions covering the old range of created_at timestamps — so migrating historical rows fails with PG::CheckViolation, because there's no partition for them to land in. The documented fix was specific: keep upgrading to 18.3.0 or later, where two follow-up migrations were said to clean up and reschedule the stuck backfill automatically. And the advice came with an explicit exception carved out: normally you shouldn't upgrade past a version with unfinished background migrations — but this specific case was called out as safe to do anyway.
Confirming the Risk Was Real, Not Theoretical
SELECT count(*) FROM sent_notifications;
-- 48,422
Not an empty table that could simply be ignored — 48,422 real rows meant this migration had actual work to do, and the partition-key collision condition genuinely applied here.
One more detail worth digging into before Station 5 started: sent_notifications was, at the time, an ordinary table, not yet partitioned — the partitioned version the forum reports referenced didn't exist yet; it would only get created during the 18.2.8 upgrade itself. The table didn't even have a created_at column yet, either, even though the documented root cause was specifically about a created_at-based partition range. That gap made sense once connected: 18.2.8 was about to add that column and partition on it — which is exactly why older rows, created before the column existed, wouldn't have a matching partition to land in.
Station 5: It Failed Exactly as Predicted
BackfillSentNotificationsAfterPartition — 48,422 rows
5 of 5 batches: failed (retried 1–3 times each, then status → 4/failed)
No surprises here — the documented risk played out precisely as described. The call, per the documented advice, was to leave it alone rather than intervene, and continue toward Station 6.

The result held up under verification. The explanation for it didn’t.
Under the Hood
+---------------------------+------------------------------------------------+---------------------------------------------------+
| Item | What I Did | Why It Mattered |
+---------------------------+------------------------------------------------+---------------------------------------------------+
| Pre-upgrade research | Found the documented failure and its | Turned a surprise failure into an expected, |
| | counterintuitive "don't fix it" advice | already-understood one |
+---------------------------+------------------------------------------------+---------------------------------------------------+
| Impact check | Confirmed the table had real, non-empty data | Ruled out "this doesn't actually apply here" |
| | before trusting the documented risk applied | before trusting the plan |
+---------------------------+------------------------------------------------+---------------------------------------------------+
| Schema detective work | Found the table had no created_at column yet, | Connected an apparently unrelated fact to the exact |
| | but the failure was about created_at partitioning | root cause the documentation described |
+---------------------------+------------------------------------------------+---------------------------------------------------+
| Decision at Station 5 | Left the failed migration alone, continued | Trusted documented advice over the instinct to |
| | upgrading rather than stopping to patch it | patch every failure immediately |
+---------------------------+------------------------------------------------+---------------------------------------------------+
What Actually Worked
Trusting the documented, counterintuitive advice — instead of the instinct to fix every failure the moment it appears — was the right call here, and the eventual finalized status and intact row count confirm it. Verifying the table's actual state before Station 5 (real row count, missing column, not-yet-partitioned structure) is what turned a generic forum warning into a confirmed, environment-specific risk instead of an assumption.
Where This Still Falls Short
This is the part of the story that didn’t hold up, and it belongs in the article rather than quietly edited out. See the appendix below.
Appendix: Verifying My Own Notes
Before writing this article, one detail needed a final check: the specific claim that two named migrations — CleanSentNotificationsFirstBackfill and RescheduleSentNotificationsBackfill — were what actually cleaned up and rescheduled the failed backfill. That claim had never been independently verified; it was carried forward from the original documentation research and never re-checked against this environment directly. Before putting it in writing, it seemed worth confirming.
SELECT id, job_class_name, status, finished_at
FROM batched_background_migrations
WHERE job_class_name = 'BackfillSentNotificationsAfterPartition';
id | job_class_name | status | finished_at
-----+------------------------------------------+--------+-------------------------------
609 | BackfillSentNotificationsAfterPartition | 6 | 2026-09-01 04:21:04.032121+00
(1 row)
That part matched expectations exactly — status = 6 (finalized), completed September 1st.
SELECT id, job_class_name, status, finished_at
FROM batched_background_migrations
WHERE job_class_name IN ('CleanSentNotificationsFirstBackfill', 'RescheduleSentNotificationsBackfill');
id | job_class_name | status | finished_at
----+-----------------+--------+-------------
(0 rows)
Zero rows. Neither named migration exists in this table. A source-code search came back the same way:
docker exec -u git -w /home/git/gitlab gitlab_gitlab_1 \
grep -rl "CleanSentNotificationsFirstBackfill\|RescheduleSentNotificationsBackfill" db/
(no output — zero matches)
Whatever actually resolved this migration, it wasn’t two independently named, identifiable migrations by those class names — at least not findable through either the database or the current source tree.
The data itself told a cleaner story:
SELECT tablename FROM pg_tables WHERE tablename LIKE '%sent_notification%';
tablename
------------------------
p_sent_notifications
p_sent_notifications_1
p_sent_notifications_2
notification_settings
(4 rows)
SELECT count(*) FROM p_sent_notifications;
-- 49478
The original sent_notifications table is gone. In its place: p_sent_notifications, properly partitioned across two sub-tables, holding 49,478 rows — slightly more than the 48,422 counted before Station 5, consistent with normal use in the time since. No data was lost. The problem is, by every measure available, actually resolved.
What can’t be confirmed is the mechanism. Three explanations remain open, and none of them can be ruled in or out from here: the migration names might simply be remembered incorrectly; they might be standard Rails schema migrations rather than batched background migrations, which wouldn’t appear in this table at all; or the fix might have been folded into some other, differently-named migration entirely. The honest version of this story has two halves — one fully confirmed, one that stops at “I don’t know, and I checked.”
Have you ever gone back to verify a detail you were confident about, only to find the outcome was real but your explanation for it wasn’t?
This is Part 9 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 4: The Day Before the Six Minutes | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 6: The Bug GitLab Hasn't Fixed Yet
GitHub: jasonChen0604
Portfolio: jason-chen-1cb56.web.app
LinkedIn: jason-cj-chen