JC_

September 15, 2026

Station 11, Part 2: 202, Not 400 — The Image That Wouldn't Die, Finally Does

Gitlab
DevOps
Postgresql
Site Reliability
Software Engineering

Live Wire: A Solo GitLab Upgrade Log — 58,818 failed requests, back in August. One API call, one month later, that finally returns what it was always supposed to.

The same delete call this series opened a mystery with, finally closing it

This is Part 16 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 11, Part 1: Three Blockers a Dry-Run Almost Missed | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Discipline, Not Luck

TL;DR

Back in article five of this series, a container image refused to delete — 58,818 failed requests traced to a protocol-level mismatch between GitLab and the registry image running underneath it. The workaround stopped the bleeding. The real fix waited for this station. With every dry-run blocker from Part 1 already resolved, the actual cutover verified the same one indicator — can a delete-by-tag call actually succeed — at three increasing levels of realism, before confirming it against the real production endpoint. The exact API call that returned 400 "digest invalid" in August now returns 202 Accepted. The cutover itself took 17 minutes against an estimated 30 to 60.

A Cliffhanger From Early in This Series

Article five of this series ended without a resolution, on purpose: a stuck deletion, traced down to a registry image that answered a capability probe honestly by its own logic, while genuinely not supporting the operation GitLab inferred from that answer. The fix — swapping the registry image itself for GitLab’s official fork — was deliberately deferred until every other station in this project had already landed safely. This is that station.

The Fix, in One Sentence

Verify the exact same indicator — can this registry actually delete by tag name — at three increasing levels of realism, and don’t call it fixed until the real production endpoint confirms it directly.

Quick Start: One Indicator, Tracked Three Times

# The exact capability check that lied back in August
curl -X OPTIONS https://<registry-host>/v2/<repo>/manifests/<tag>
# Looking specifically at the Allow header this time, not just the status code

Every verification step below asks this same question, at increasing levels of realism, until there’s nothing left to doubt.

Layer 1: The Protocol Check, Done Right This Time

Against an isolated mini registry — copied, not hard-linked, so nothing about this test could touch real data — the same OPTIONS probe that gave a false positive against the old image gave an honest answer against the fork:

Allow: DELETE, GET, HEAD, PUT

Layer 2: Real Data, Still Fully Isolated

The isolated registry wasn’t synthetic — it held a genuine copy of one real project’s repositories and every blob they referenced, roughly 1.5GB, completely separated from production. Against that real data:

DELETE by tag name  →  202 Accepted
tags/list confirms the tag now returns null
push (full blob-upload flow)  →  201 Created

Both directions worked — deleting what should be deletable, and confirming new content could still be pushed afterward.

Same indicator, three levels of realism, then the real thing

Layer 3: The Real Endpoint, the Real Project

TOKEN=$(curl ... "/jwt/auth?...scope=...delete")
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://<host>:5000/v2/tqms/alpha/alpha-ai-agent/manifests/DEV"
HTTP/2 202
tags_now=""

Two minutes of registry logs afterward: zero occurrences of “digest invalid.” Not reduced. Zero.

The Exact Comparison This Series Has Been Waiting For

2026-08-21 (stop-the-bleeding diagnosis)
DELETE /v2/tqms/alpha/alpha-ai-agent/manifests/qas
→ 400 "digest invalid"
2026-09-15 (real fix, same project)
DELETE /v2/tqms/alpha/alpha-ai-agent/manifests/DEV
→ 202 Accepted

Same project. Same shape of API call. One month apart. One returns the error this whole thread started with — the other returns exactly what it was always supposed to.

The Cutover Itself: 17 Minutes

12:01 → 12:18

17 minutes, against an estimated 30 to 60. This is the same pattern this series has traced before, back at Station 4: the real risk gets spent the day before, in a dry-run, not during the maintenance window itself. Every blocker from Part 1 — the missing config, the database detection bug, the ownership mismatch, the health check quirk — was already resolved before this window opened. What was left to do during the window itself was small by design.

Closing the Backlog: 3,098 Jobs, Cleared

With the registry fix confirmed working, the deferred decision from “Two Thousand Stalled Jobs” finally had an answer:

Sidekiq::DeadSet.new.clear
# before=3098 after=0

The dead queue got cleared outright, not retried job by job. The reasoning: these are job execution records, not business data — retrying them would have meant re-attempting deletions that had already been abandoned months ago, on repositories that may have moved on since. If a real cleanup need exists going forward, it’s simpler to handle it directly than to try to resurrect a six-month-old backlog.

Under the Hood

+---------------------------+------------------------------------------------+----------------------------------------------------+
| Layer                       | What Was Tested                                   | Result                                                |
+---------------------------+------------------------------------------------+----------------------------------------------------+
| Protocol                    | OPTIONS capability probe, isolated registry        | Allow: DELETE, GET, HEAD, PUT — honest this time      |
+---------------------------+------------------------------------------------+----------------------------------------------------+
| Isolated real data            | Delete by tag + full push flow, real repo copy     | 202 on delete, 201 on push — both directions work     |
+---------------------------+------------------------------------------------+----------------------------------------------------+
| Production                   | The exact delete call from the August diagnosis    | 202, zero digest invalid errors in the two            |
|                             | re-run against the real endpoint                    | minutes after                                          |
+---------------------------+------------------------------------------------+----------------------------------------------------+
| Backlog closure               | Cleared 3,098 dead jobs once the fix was confirmed  | Chose clearing over retrying — the underlying          |
|                             |                                                    | deletions were already long abandoned                  |
+---------------------------+------------------------------------------------+----------------------------------------------------+

What Actually Worked

Testing the exact same indicator at three increasing levels of realism — protocol, isolated real data, then production — is what turned “the dry-run problems are fixed” into actual confidence that the core bug itself was gone, not just the blockers around it. Comparing the same API call, on the same project, directly against the original August failure is the clearest possible proof this series could offer that the fix is real rather than assumed.

Where This Still Falls Short

This closes the loop this series opened in article five, and it closes Station 11 — but it doesn’t close the series. What’s left is a look back across all eleven stations, and whatever pattern holds across nearly all of them once the adrenaline of any single station wears off.

If you’ve ever carried an open question across months of unrelated work, what did it feel like the day you finally got to close it?

This is Part 16 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Station 11, Part 1: Three Blockers a Dry-Run Almost Missed | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Discipline, Not Luck