JC_

September 13, 2026

Two Thousand Stalled Jobs

Gitlab
DevOps
Postgresql
Site Reliability
Software Engineering

Live Wire: A Solo GitLab Upgrade Log — Ten stations into this upgrade, an unrelated number showed up in the admin panel: over 2,800 dead jobs, the oldest one six months old.

A number that had nothing to do with the upgrade, and everything to do with a bug this series already met once

This is Part 14 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Reaching 19.2.4: The Milestone | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 11, Part 1: Three Blockers a Dry-Run Almost Missed

TL;DR

Ten stations into this upgrade, a routine glance at the admin panel’s Background Jobs page turned up something unrelated to any of it: a dead-job queue sitting at 2,887 entries, spanning six months of accumulation. Classifying them by job type found that 89.9% traced directly back to a bug this series already met once — the Container Registry protocol mismatch from early in this project. A workaround stopping the bleeding six months ago never made the underlying failures disappear; it just made them pile up somewhere quieter. The decision here isn’t a fix — it’s choosing to wait until the actual root cause is resolved before deciding what to do with the backlog it left behind.

A Number That Didn’t Belong to This Upgrade

Ten stations of version bumps, migrations, and database work — and the thing that stood out wasn’t any of that. A routine check of the admin panel’s Background Jobs page showed a dead-job queue that had grown well past two thousand entries, with no obvious connection to anything this series had been actively working on.

The Fix, in One Sentence

Classify before concluding — a large, vague number becomes a specific, already-understood problem the moment it’s broken down by what’s actually failing.

Quick Start: Sizing Up a Dead Job Queue

stats = Sidekiq::Stats.new
puts "processed=#{stats.processed} failed=#{stats.failed} enqueued=#{stats.enqueued} " \
     "scheduled_size=#{stats.scheduled_size} retry_size=#{stats.retry_size} dead_size=#{stats.dead_size}"
processed=874164054 failed=26787 enqueued=0 scheduled_size=19 retry_size=0 dead_size=2887
klass_count = Hash.new(0)
Sidekiq::DeadSet.new.each { |job| klass_count[job.klass] += 1 }

The first command gives the scale. The second one gives the story.

2,887 Jobs, Six Months Deep

oldest: 2026-03-12 06:00:05 +0000
newest: 2026-09-08 05:17:08 +0000

Not a recent spike — a backlog that had been quietly accumulating for roughly six months, untouched, without anyone flagging it as a problem worth investigating until it happened to be glanced at during an unrelated upgrade project.

A vague number becomes a specific, already-understood cause the moment it’s classified

Classifying the Backlog

Ci::DeleteObjectsWorker                                        2,279
ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper              253
ContainerExpirationPolicies::CleanupContainerRepositoryWorker     222
ContainerRegistry::DeleteContainerRepositoryWorker                 96
Database::BatchedBackgroundMigration::MainExecutionWorker          15
(remaining classes, 1–12 jobs each)

2,279 + 222 + 96 = 2,597 jobs — 89.9% of the entire dead queue — all belonging to job classes directly tied to deleting or cleaning up container registry images.

Recognizing an Old Friend

This is the same failure this series already documented, back in “The Image That Wouldn’t Die.” GitLab attempting to delete registry images or run cleanup policies, failing repeatedly against the same protocol-level mismatch, retrying, and eventually landing in the dead queue once retries were exhausted. This isn’t a new problem, and it isn’t something the upgrade caused. It’s the visible, accumulating residue of a bug that was stopped from causing active harm six months ago, but never actually resolved — the fix for that is exactly what Station 11 exists to deliver.

The Decision: Not Yet

No action gets taken on this backlog right now. The plan explicitly defers the choice — batch-retry the whole set, or simply clear the dead queue outright — until after Station 11 replaces the registry image and the real fix is confirmed working. Retrying now, against a registry still carrying the same mismatch, would just recreate the same 2,597 failures a second time. That decision is written directly into the closing checklist for Station 11, specifically so it doesn’t get quietly forgotten once the registry swap itself is done.

Under the Hood

+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Item                        | What I Found                                     | Why It Mattered                                       |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Scale                       | 2,887 dead jobs, spanning roughly six months      | Turned a vague admin-panel number into a              |
|                             |                                                    | measurable, dated backlog                              |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Classification               | 89.9% traced to registry deletion/cleanup          | Converted an unexplained pile of jobs into a           |
|                             | job classes                                       | specific, already-understood root cause                 |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Connection                   | Matched directly to the protocol mismatch from    | Confirmed this wasn't new — it was the same bug's      |
|                             | "The Image That Wouldn't Die"                     | ongoing, uncounted cost                                  |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Decision                     | Deferred retry/clear decision to after Station    | Avoided recreating the same failures against a         |
|                             | 11's registry fix, logged in the closing checklist | registry that still had the underlying bug              |
+---------------------------+-----------------------------------------------+----------------------------------------------------+

What Actually Worked

Classifying by job class before drawing any conclusion is what turned “there are 2,887 mystery jobs in here” into “89.9% of these are one already-known problem” — the difference between those two framings is the difference between a vague worry and an actionable, already-understood fact. Logging the deferred decision directly into Station 11’s own closing checklist, rather than trusting memory to carry it forward, is a small habit that matters exactly because it’s easy to forget once the more dramatic part of a station is finished.

Where This Still Falls Short

Nothing gets fixed in this piece, and it’s worth being direct about that: this is a diagnosis and a deliberate decision to wait, not a resolution. The remaining 10.1% — including 253 jobs wrapped generically as ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper, without further digging into what specific work those actually represent — never got investigated further here. A workaround that stops active harm isn't the same as a fix, and this backlog is the receipt for that difference: six months of quiet accumulation, for a bug that was "handled" back in August.

Have you ever gone looking for one thing and found the long-delayed bill for a workaround you thought had already closed the book on something?

This is Part 14 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Reaching 19.2.4: The Milestone | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 11, Part 1: Three Blockers a Dry-Run Almost Missed